In-field illumination and imaging for eye tracking

ABSTRACT

Disclosed herein are techniques for eye tracking in near-eye display devices. In some embodiments, an illuminator for eye tracking is provided. The illuminator includes a light source configured to be positioned within a field of view of an eye of a user; a first reflector configured to shadow the light source from a field of view of a camera; and a second reflector configured to receive light from the light source that is reflected by the eye of the user, and to direct the light toward the camera.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/033,099, filed on Jul. 11, 2018, which claims priority under 35U.S.C. § 119 to U.S. Provisional Patent Application No. 62/675,650,filed on May 23, 2018, the contents of both of which are herebyincorporated by reference in their entireties for all purposes.

BACKGROUND

An artificial reality system generally includes a display panelconfigured to present artificial images that depict objects in a virtualenvironment. The display panel may display virtual objects or combinereal objects with virtual objects, as in virtual reality (VR), augmentedreality (AR), or mixed reality (MR) applications. To interact with theartificial reality system, a user may need to provide inputs directed toat least a portion of the displayed image. Some artificial realitysystems may include a dedicated input/output interface for receivinguser inputs, such as hand and/or finger movements. However, traditionalinput/output interfaces may require frequent and active user inputs, andthus may prevent the user from having a fully immersive experience inthe artificial reality environment.

An eye-tracking system can track the gaze of an artificial reality(e.g., VR/AR/MR) system so that the artificial reality system knowswhere the user is looking, and thus can provide a more immersiveinterface than a typical input/output interface predominantly reliant ona handheld peripheral input/output device. Eye-tracking may also be usedfor foveated imaging, foveated transmission of image data, alertnessmonitoring, etc. Existing eye-tracking systems may use light sources(e.g., infrared light) positioned at the periphery of the user's fieldof view to illuminate the eye, where the light illuminating the eye maybe reflected specularly by the cornea of the user's eye, resulting in“glints” in a captured image of the eye. The position (e.g., gazedirection or rotation position) of the eye may be determined based on,for example, the location of the glints relative to a known feature ofthe eye (e.g., center of the pupil) in the captured image. Existingeye-tracking systems may also use imaging systems (e.g., cameras) tocapture the light reflected by various surfaces of the eye. The camerasmay also be positioned at the periphery of the user's field of view.

There may be several issues associated with existing eye-trackingtechnologies. One of the issues is the size of the glints in thecaptured image for a light source that may not be a “point source.” Forexample, an LED that may be used as the light source may have anemission area with a linear dimension of 200 μm or more. Thus, when thewhole LED emission area is captured, the glint may not appear as a pointin the captured image. Consequently, the center location of the glint inthe image may not be precisely determined, and the errors in theapproximation may lead to errors in the eye-tracking result. Further,the peripheral location of the light sources may negatively impact theaccuracy of the eye-tracking due to, for example, the large angles ofthe illuminating light from the light sources to the eye. While in-fieldillumination may offer greater accuracy, in-field illumination may haveseveral challenges. For example, light sources that are positionedwithin the user's field of view may affect the quality of thesee-through real-world images and the displayed images. Further, thecameras that are positioned at the periphery of the user's field of viewmay observe the eye from large angles, thereby reducing the accuracy ofthe eye-tracking computations. Further, observing the eye fromperipheral locations may increase the likelihood that the camera's viewof the eye may be obstructed by facial features such as eyelids,eyelashes, etc.

SUMMARY

The present disclosure generally relates to eye tracking in near-eyedisplay devices. In some embodiments, an illuminator for eye tracking isprovided. The illuminator includes a light source configured to bepositioned within a field of view of an eye of a user; a first reflectorconfigured to shadow the light source from a field of view of a camera;and a second reflector configured to receive light from the light sourcethat is reflected by the eye of the user, and to direct the light towardthe camera.

The first reflector may be a first coating on a first prism, and thesecond reflector may be a second coating on a second prism. A firstportion of the second prism that is shadowed by the first reflector maybe uncoated, and a second portion of the second prism that is unshadowedby the first reflector may be coated by the first coating. The lightsource may be configured to emit light that propagates between the firstreflector and the second reflector. Each of the first reflector and thesecond reflector may be configured to reflect infrared light and totransmit visible light.

The illuminator may also include a substrate having a first surface onwhich the light source is mounted and a second surface through whichlight is outcoupled toward the eye of the user. The first reflector andthe second reflector may be arranged within the substrate between thefirst surface and the second surface. In addition, the illuminator mayinclude a beam diverting component configured to direct light from thelight source toward the eye of the user.

The beam diverting component may be formed on the second surface of thesubstrate, indented toward the first surface of the substrate, and havea shape of a prism, a cone, a diffraction grating, or a lens.Alternatively, the beam diverting component may be formed on the secondsurface of the substrate, protrude away from the first surface of thesubstrate, and have a shape of a prism or a cone. As anotheralternative, the beam diverting component may be a surface reliefgrating that is formed at the second surface of the substrate or avolume Bragg grating.

In other embodiments, the beam diverting component may include a thirdreflector and a fourth reflector, each of which is arranged within thesubstrate between the first surface and the second surface. The thirdreflector may be configured to reflect light from the light source tothe fourth reflector, and the fourth reflector may be configured toreflect the light toward the second surface of the substrate.

According to another aspect, a system for eye tracking is provided. Thesystem may include any of the configurations of the illuminatordescribed above, along with a camera configured to capture an image ofthe light source reflected by the eye of the user.

This summary is neither intended to identify key or essential featuresof the claimed subject matter, nor is it intended to be used inisolation to determine the scope of the claimed subject matter. Thesubject matter should be understood by reference to appropriate portionsof the entire specification of this disclosure, any or all drawings, andeach claim. The foregoing, together with other features and examples,will be described in more detail below in the following specification,claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments are described in detail below with reference tothe following figures:

FIG. 1 is a simplified block diagram of an example artificial realitysystem environment including a near-eye display, according to certainembodiments;

FIG. 2 is a perspective view of a simplified example near-eye displayincluding various sensors;

FIG. 3 is a perspective view of a simplified example near-eye displayincluding an example eye-tracking unit;

FIG. 4 is a cross-sectional view of an example near-eye displayincluding an example eye-tracking unit;

FIG. 5 illustrates light reflections and diffusions by an eye during eyetracking;

FIG. 6 is a simplified diagram of an example system for eye tracking inan example near-eye display, according to certain embodiments;

FIG. 7 is a simplified diagram of an example illumination system for eyetracking in an example near-eye display, according to certainembodiments;

FIGS. 8A-8C are simplified diagrams of example illumination systems foreye tracking in example near-eye displays, according to certainembodiments;

FIGS. 9A-9C are simplified diagrams of example illumination systems foreye tracking in example near-eye displays, according to certainembodiments;

FIGS. 10A-10C are simplified diagrams of example illumination systemsfor eye tracking in example near-eye displays, according to certainembodiments;

FIG. 11 is a flow chart illustrating an example method of eyeillumination for eye tracking in a near-eye display, according tocertain embodiments;

FIG. 12 is a perspective view of an example near-eye display in the formof a head-mounted display (HMD) device for implementing some of theexamples disclosed herein; and

FIG. 13 is a simplified block diagram of an example electronic system ofan example near-eye display for implementing some of the examplesdisclosed herein.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, specificdetails are set forth in order to provide a thorough understanding ofexamples of the disclosure. However, it will be apparent that variousexamples may be practiced without these specific details. For example,devices, systems, structures, assemblies, methods, and other componentsmay be shown as components in block diagram form in order not to obscurethe examples in unnecessary detail. In other instances, well-knowndevices, processes, systems, structures, and techniques may be shownwithout necessary detail in order to avoid obscuring the examples. Thefigures and description are not intended to be restrictive. The termsand expressions that have been employed in this disclosure are used asterms of description and not of limitation, and there is no intention inthe use of such terms and expressions of excluding any equivalents ofthe features shown and described or portions thereof.

As used herein, visible light may refer to light with a wavelengthbetween about 400 nm and about 750 nm. Near infrared (NIR) light mayrefer to light with a wavelength between about 750 nm and about 2500 nm.The desired infrared (IR) wavelength range may refer to the wavelengthrange of IR light that can be detected by a suitable IR sensor (e.g., acomplementary metal-oxide semiconductor (CMOS) or a charge-coupleddevice (CCD) sensor), such as between 830 nm and 860 nm or between 930nm and 980 nm.

As also used herein, a substrate may refer to a medium within which anarray of chirped gratings may be inscribed. A chirped grating may referto a grating whose pitch and angle of orientation changes over theextent of the grating. The substrate may include one or more types ofdielectric materials, such as glass, quartz, plastic, polymer,poly(methyl methacrylate) (PMMA), crystal, or ceramic. At least one typeof material of the substrate may be transparent to visible light and MRlight. A thickness of the substrate may range from, for example, lessthan about 1 mm to less than about 10 mm. As used herein, a material maybe “transparent” to a light beam if the light beam can pass through thematerial with a high transmission rate, such as larger than 60%, 75%,80%, 90%, 95%, 98%, 99%, or higher, where a small portion of the lightbeam (e.g., less than 40%, 25%, 20%, 10%, 5%, 2%, 1%, or less) may bescattered, reflected, or absorbed by the material. The transmission rate(i.e., transmissivity) may be represented by either a photopicallyweighted or an unweighted average transmission rate over a range ofwavelengths, or the lowest transmission rate over a range ofwavelengths, such as the visible wavelength range.

An artificial reality system, such as a virtual reality (VR), augmentedreality (AR), or mixed reality (MR) system, may include a near-eyedisplay (e.g., a headset or a pair of glasses) configured to presentcontent to a user via an electronic or optic display and, in some cases,may also include a console configured to generate content forpresentation to the user and to provide the generated content to thenear-eye display for presentation. To improve user interaction withpresented content, the console may modify or generate content based on alocation where the user is looking, which may be determined by trackingthe user's eye. Tracking the eye may include tracking the positionand/or shape of the pupil of the eye, and/or the rotational position(gaze direction) of the eye. To track the eye, the near-eye display mayilluminate a surface of the user's eye using light sources mounted to orwithin the near-eye display, according to at least one embodiment. Animaging device (e.g., a camera) included in the vicinity of the near-eyedisplay may then capture light reflected by various surfaces of theuser's eye. Light that is reflected specularly off the cornea of theuser's eye may result in “glints” in the captured image. One way toilluminate the eye to see the pupil as well as the glints is to use atwo-dimensional (2D) array of light-emitting diodes (LEDs). According toembodiments of the invention, these LEDs may be placed within the user'sfield of view. Techniques such as a centroiding algorithm may be used toaccurately determine the locations of the glints on the eye in thecaptured image, and the rotational position (e.g., the gaze direction)of the eye may then be determined based on the locations of the glintsrelative to a known feature of the eye (e.g., the center of the pupil)within the captured image.

Positioning a single light source or a plurality of light sources withinthe user's field of view may offer greater eye-tracking accuracy thanpositioning the light sources at the periphery of the user's field ofview. For example, the probability of capturing glints over all gazeangles of the eye is higher when the light sources are located withinthe user's field of view. Further, the light sources may be configuredsuch that they are effectively invisible to the user. This may beaccomplished by using light sources with a very small form factor, suchas less than 500 nm, less than 400 nm, or less than 200 μm. The formfactor may refer to a maximum linear dimension of the light source in aplane that is parallel to an emission surface of the light source.

A form factor of 200 μm may be a lower limit of what an eye is able toresolve. Alternatively, a form factor of 200 μm may be an upper limit ofwhat is bothersome to the user when the light source is within theuser's field of view. For example, a light source with a form factor of200 μm may appear similar to a dust speck on the user's glasses, and maynot interfere with the user's vision through the glasses. Some examplesof light sources with a very small form factor are vertical cavitysurface emitting lasers (VCSELs) that have a bare die size of less than160 μm and an emission cone with an angle of less than 25°, andmicro-LEDs that have a bare die size of less than 200 μm and an emissioncone with an angle of less than 30°. The die size may refer to a lineardimension of the VCSEL or the micro-LED in a plane that is parallel toan emission surface of the VCSEL or the micro-LED. For example, theVCSEL or the micro-LED may have a square shape within the plane that isparallel to the emission surface, such that each of the sides of thesquare has a linear dimension of less than 200 μm. Further, positioningthe light sources within the user's field of view may offer greaterflexibility in the placement and distribution of the light sources, suchthat the amount of light captured by the camera is maximized. Althoughthe eye may be able to detect near infrared light from a light sourcethat is very bright, the light sources may be operated at lowerbrightness levels to minimize this effect.

FIG. 1 is a simplified block diagram of an example artificial realitysystem environment 100 including a near-eye display 120, in accordancewith certain embodiments. Artificial reality system environment 100shown in FIG. 1 may include a near-eye display 120, an external imagingdevice 150, and an input/output interface 140 that are each coupled to aconsole 110. While FIG. 1 shows example artificial reality systemenvironment 100 including one near-eye display 120, one external imagingdevice 150, and one input/output interface 140, any number of thesecomponents may be included in artificial reality system environment 100,or any of the components may be omitted. For example, there may bemultiple near-eye displays 120 monitored by one or more external imagingdevices 150 in communication with console 110. In alternativeconfigurations, different or additional components may be included inartificial reality system environment 100.

Near-eye display 120 may be a head-mounted display that presents contentto a user. Examples of content presented by near-eye display 120 includeone or more of images, videos, audios, or some combination thereof. Insome embodiments, audio may be presented via an external device (e.g.,speakers and/or headphones) that receives audio information fromnear-eye display 120, console 110, or both, and presents audio databased on the audio information. Near-eye display 120 may include one ormore rigid bodies, which may be rigidly or non-rigidly coupled to eachother. A rigid coupling between rigid bodies may cause the coupled rigidbodies to act as a single rigid entity. A non-rigid coupling betweenrigid bodies may allow the rigid bodies to move relative to each other.In various embodiments, near-eye display 120 may be implemented in anysuitable form factor, including a pair of glasses. Additionally, invarious embodiments, the functionality described herein may be used in aheadset that combines images of an environment external to near-eyedisplay 120 and content received from console 110, or from any otherconsole generating and providing content to a user. Therefore, near-eyedisplay 120, and methods for eye tracking described herein, may augmentimages of a physical, real-world environment external to near-eyedisplay 120 with generated content (e.g., images, video, sound, etc.) topresent an augmented reality to a user.

In various embodiments, near-eye display 120 may include one or more ofdisplay electronics 122, display optics 124, one or more locators 126,one or more position sensors 128, an eye-tracking unit 130, and aninertial measurement unit (IMU) 132. Near-eye display 120 may omit anyof these elements or include additional elements in various embodiments.Additionally, in some embodiments, near-eye display 120 may includeelements combining the function of various elements described inconjunction with FIG. 1.

Display electronics 122 may display images to the user according to datareceived from console 110. In various embodiments, display electronics122 may include one or more display panels, such as a liquid crystaldisplay (LCD), an organic light emitting diode (OLED) display, amicro-LED display, an active-matrix OLED display (AMOLED), a transparentOLED display (TOLED), or some other display. For example, in oneimplementation of near-eye display 120, display electronics 122 mayinclude a front TOLED panel, a rear display panel, and an opticalcomponent (e.g., an attenuator, polarizer, or diffractive or spectralfilm) between the front and rear display panels. Display electronics 122may include sub-pixels to emit light of a predominant color such as red,green, blue, white, or yellow. In some implementations, displayelectronics 122 may display a 3D image through stereo effects producedby two-dimensional panels to create a subjective perception of imagedepth. For example, display electronics 122 may include a left displayand a right display positioned in front of a user's left eye and righteye, respectively. The left and right displays may present copies of animage shifted horizontally relative to each other to create astereoscopic effect (i.e., a perception of image depth by a user viewingthe image).

In certain embodiments, display optics 124 may display image contentoptically (e.g., using optical waveguides and couplers), or magnifyimage light received from display electronics 122, correct opticalerrors associated with the image light, and present the corrected imagelight to a user of near-eye display 120. In various embodiments, displayoptics 124 may include one or more optical elements. Example opticalelements may include a substrate, optical waveguides, an aperture, aFresnel lens, a convex lens, a concave lens, a filter, or any othersuitable optical element that may affect image light emitted fromdisplay electronics 122. Display optics 124 may include a combination ofdifferent optical elements as well as mechanical couplings to maintainrelative spacing and orientation of the optical elements in thecombination. One or more optical elements in display optics 124 may havean optical coating, such as an anti-reflective coating, a reflectivecoating, a filtering coating, or a combination of different opticalcoatings.

Magnification of the image light by display optics 124 may allow displayelectronics 122 to be physically smaller, weigh less, and consume lesspower than larger displays. Additionally, magnification may increase afield of view of the displayed content. In some embodiments, displayoptics 124 may have an effective focal length larger than the spacingbetween display optics 124 and display electronics 122 to magnify imagelight projected by display electronics 122. The amount of magnificationof image light by display optics 124 may be adjusted by adding orremoving optical elements from display optics 124.

Display optics 124 may be designed to correct one or more types ofoptical errors, such as two-dimensional optical errors,three-dimensional optical errors, or a combination thereof.Two-dimensional errors may include optical aberrations that occur in twodimensions. Example types of two-dimensional errors may include barreldistortion, pincushion distortion, longitudinal chromatic aberration,and transverse chromatic aberration. Three-dimensional errors mayinclude optical errors that occur in three dimensions. Example types ofthree-dimensional errors may include spherical aberration, comaticaberration, field curvature, and astigmatism. In some embodiments,content provided to display electronics 122 for display may bepre-distorted, and display optics 124 may correct the distortion when itreceives image light from display electronics 122 generated based on thepre-distorted content.

Locators 126 may be objects located in specific positions on near-eyedisplay 120 relative to one another and relative to a reference point onnear-eye display 120. Console 110 may identify locators 126 in imagescaptured by external imaging device 150 to determine the artificialreality headset's position, orientation, or both. A locator 126 may be alight emitting diode (LED), a corner cube reflector, a reflectivemarker, a type of light source that contrasts with an environment inwhich near-eye display 120 operates, or some combinations thereof. Inembodiments where locators 126 are active components (e.g., LEDs orother types of light emitting devices), locators 126 may emit light inthe visible band (e.g., about 380 nm to 750 nm), in the infrared (IR)band (e.g., about 750 nm to 1 mm), in the ultraviolet band (e.g., about10 nm to about 380 nm), in another portion of the electromagneticspectrum, or in any combination of portions of the electromagneticspectrum.

In some embodiments, locators 126 may be located beneath an outersurface of near-eye display 120. A portion of near-eye display 120between a locator 126 and an entity external to near-eye display 120(e.g., external imaging device 150, a user viewing the outer surface ofnear-eye display 120) may be transparent to the wavelengths of lightemitted or reflected by locators 126 or is thin enough to notsubstantially attenuate the light emitted or reflected by locators 126.In some embodiments, the outer surface or other portions of near-eyedisplay 120 may be opaque in the visible band, but is transparent in theIR band, and locators 126 may be under the outer surface and may emitlight in the IR band.

External imaging device 150 may generate slow calibration data based oncalibration parameters received from console 110. Slow calibration datamay include one or more images showing observed positions of locators126 that are detectable by external imaging device 150. External imagingdevice 150 may include one or more cameras, one or more video cameras,any other device capable of capturing images including one or more oflocators 126, or some combinations thereof. Additionally, externalimaging device 150 may include one or more filters (e.g., to increasesignal to noise ratio). External imaging device 150 may be configured todetect light emitted or reflected from locators 126 in a field of viewof external imaging device 150. In embodiments where locators 126include passive elements (e.g., retroreflectors), external imagingdevice 150 may include a light source that illuminates some or all oflocators 126, which may retro-reflect the light to the light source inexternal imaging device 150. Slow calibration data may be communicatedfrom external imaging device 150 to console 110, and external imagingdevice 150 may receive one or more calibration parameters from console110 to adjust one or more imaging parameters (e.g., focal length, focus,frame rate, sensor temperature, shutter speed, aperture, etc.).

Position sensors 128 may generate one or more measurement signals inresponse to motion of near-eye display 120. Examples of position sensors128 may include accelerometers, gyroscopes, magnetometers, othermotion-detecting or error-correcting sensors, or some combinationsthereof. For example, in some embodiments, position sensors 128 mayinclude multiple accelerometers to measure translational motion (e.g.,forward/back, up/down, or left/right) and multiple gyroscopes to measurerotational motion (e.g., pitch, yaw, or roll). In some embodiments,various position sensors may be oriented orthogonally to each other.

IMU 132 may be an electronic device that generates fast calibration databased on measurement signals received from one or more of positionsensors 128. Position sensors 128 may be located external to IMU 132,internal to IMU 132, or some combination thereof. Based on the one ormore measurement signals from one or more position sensors 128, IMU 132may generate fast calibration data indicating an estimated position ofnear-eye display 120 relative to an initial position of near-eye display120. For example, IMU 132 may integrate measurement signals receivedfrom accelerometers over time to estimate a velocity vector andintegrate the velocity vector over time to determine an estimatedposition of a reference point on near-eye display 120. Alternatively,IMU 132 may provide the sampled measurement signals to console 110,which may determine the fast calibration data. While the reference pointmay generally be defined as a point in space, in various embodiments,the reference point may also be defined as a point within near-eyedisplay 120 (e.g., a center of IMU 132).

Eye-tracking unit 130 may include one or more imaging devices configuredto capture eye tracking data, which an eye-tracking module 118 inconsole 110 may use to track the user's eye. Eye tracking data may referto data output by eye-tracking unit 130. Example eye tracking data mayinclude images captured by eye-tracking unit 130 or information derivedfrom the images captured by eye-tracking unit 130. Eye tracking mayrefer to determining an eye's position, including orientation andlocation of the eye, relative to near-eye display 120. For example,eye-tracking module 118 may output the eye's pitch and yaw based onimages of the eye captured by eye-tracking unit 130. In variousembodiments, eye-tracking unit 130 may measure electromagnetic energyreflected by the eye and communicate the measured electromagnetic energyto eye-tracking module 118, which may then determine the eye's positionbased on the measured electromagnetic energy. For example, eye-trackingunit 130 may measure electromagnetic waves such as visible light,infrared light, radio waves, microwaves, waves in any other part of theelectromagnetic spectrum, or a combination thereof reflected by an eyeof a user.

Eye-tracking unit 130 may include one or more eye-tracking systems. Aneye-tracking system may include an imaging system to image one or moreeyes and may optionally include a light emitter, which may generatelight that is directed to an eye such that light reflected by the eyemay be captured by the imaging system. For example, eye-tracking unit130 may include a coherent light source (e.g., a VCSEL) emitting lightin the visible spectrum or infrared spectrum, and a camera capturing thelight reflected by the user's eye. As another example, eye-tracking unit130 may capture reflected radio waves emitted by a miniature radar unit.Eye-tracking unit 130 may use low-power light emitters that emit lightat frequencies and intensities that would not injure the eye or causephysical discomfort. Eye-tracking unit 130 may be arranged to increasecontrast in images of an eye captured by eye-tracking unit 130 whilereducing the overall power consumed by eye-tracking unit 130 (e.g.,reducing power consumed by a light emitter and an imaging systemincluded in eye-tracking unit 130). For example, in someimplementations, eye-tracking unit 130 may consume less than 100milliwatts of power.

In some embodiments, eye-tracking unit 130 may include one light emitterand one camera to track each of the user's eyes. In other embodiments,eye-tracking unit 130 may include a plurality of light emitters and onecamera to track each of the user's eyes. Eye-tracking unit 130 may alsoinclude different eye-tracking systems that operate together to provideimproved eye tracking accuracy and responsiveness. For example,eye-tracking unit 130 may include a fast eye-tracking system with a fastresponse time and a slow eye-tracking system with a slower responsetime. The fast eye-tracking system may frequently measure an eye tocapture data used by eye-tracking module 118 to determine the eye'sposition relative to a reference eye position. The slow eye-trackingsystem may independently measure the eye to capture data used byeye-tracking module 118 to determine the reference eye position withoutreference to a previously determined eye position. Data captured by theslow eye-tracking system may allow eye-tracking module 118 to determinethe reference eye position with greater accuracy than the eye's positiondetermined from data captured by the fast eye-tracking system. Invarious embodiments, the slow eye-tracking system may provideeye-tracking data to eye-tracking module 118 at a lower frequency thanthe fast eye-tracking system. For example, the slow eye-tracking systemmay operate less frequently or have a slower response time to conservepower.

Eye-tracking unit 130 may be configured to estimate the orientation ofthe user's eye. The orientation of the eye may correspond to thedirection of the user's gaze within near-eye display 120. Theorientation of the user's eye may be defined as the direction of thefoveal axis, which is the axis between the fovea (an area on the retinaof the eye with the highest concentration of photoreceptors) and thecenter of the eye's pupil. In general, when a user's eyes are fixed on apoint, the foveal axes of the user's eyes intersect that point. Thepupillary axis of an eye may be defined as the axis that passes throughthe center of the pupil and is perpendicular to the corneal surface. Ingeneral, even though the pupillary axis and the foveal axis intersect atthe center of the pupil, the pupillary axis may not directly align withthe foveal axis. For example, the orientation of the foveal axis may beoffset from the pupillary axis by approximately −1° to 8° laterally andabout ±4° vertically. Because the foveal axis is defined according tothe fovea, which is located in the back of the eye, the foveal axis maybe difficult or impossible to measure directly in some eye trackingembodiments. Accordingly, in some embodiments, the orientation of thepupillary axis may be detected and the foveal axis may be estimatedbased on the detected pupillary axis.

In general, the movement of an eye corresponds not only to an angularrotation of the eye, but also to a translation of the eye, a change inthe torsion of the eye, and/or a change in the shape of the eye.Eye-tracking unit 130 may also be configured to detect the translationof the eye, which may be a change in the position of the eye relative tothe eye socket. In some embodiments, the translation of the eye may notbe detected directly, but may be approximated based on a mapping from adetected angular orientation. Translation of the eye corresponding to achange in the eye's position relative to the eye-tracking unit may alsobe detected. Translation of this type may occur, for example, due to ashift in the position of near-eye display 120 on a user's head.Eye-tracking unit 130 may also detect the torsion of the eye and therotation of the eye about the pupillary axis. Eye-tracking unit 130 mayuse the detected torsion of the eye to estimate the orientation of thefoveal axis from the pupillary axis. Eye-tracking unit 130 may alsotrack a change in the shape of the eye, which may be approximated as askew or scaling linear transform or a twisting distortion (e.g., due totorsional deformation). Eye-tracking unit 130 may estimate the fovealaxis based on some combinations of the angular orientation of thepupillary axis, the translation of the eye, the torsion of the eye, andthe current shape of the eye.

In some embodiments, eye-tracking unit 130 may include multiple emittersor at least one emitter that can project a structured light pattern onall portions or a portion of the eye. The structured light pattern maybe distorted due to the shape of the eye when viewed from an offsetangle. Eye-tracking unit 130 may also include at least one camera thatmay detect the distortions (if any) of the structured light patternprojected onto the eye. The camera may be oriented on a different axisto the eye than the emitter. By detecting the deformation of thestructured light pattern on the surface of the eye, eye-tracking unit130 may determine the shape of the portion of the eye being illuminatedby the structured light pattern. Therefore, the captured distorted lightpattern may be indicative of the 3D shape of the illuminated portion ofthe eye. The orientation of the eye may thus be derived from the 3Dshape of the illuminated portion of the eye. Eye-tracking unit 130 canalso estimate the pupillary axis, the translation of the eye, thetorsion of the eye, and the current shape of the eye based on the imageof the distorted structured light pattern captured by the camera.

Near-eye display 120 may use the orientation of the eye to, e.g.,determine an inter-pupillary distance (IPD) of the user, determine gazedirection, introduce depth cues (e.g., blur image outside of the user'smain line of sight), collect heuristics on the user interaction in theVR media (e.g., time spent on any particular subject, object, or frameas a function of exposed stimuli), some other functions that are basedin part on the orientation of at least one of the user's eyes, or somecombination thereof. Because the orientation may be determined for botheyes of the user, eye-tracking unit 130 may be able to determine wherethe user is looking. For example, determining a direction of a user'sgaze may include determining a point of convergence based on thedetermined orientations of the user's left and right eyes. A point ofconvergence may be the point where the two foveal axes of the user'seyes intersect (or the nearest point between the two axes). Thedirection of the user's gaze may be the direction of a line passingthrough the point of convergence and the mid-point between the pupils ofthe user's eyes.

Input/output interface 140 may be a device that allows a user to sendaction requests to console 110. An action request may be a request toperform a particular action. For example, an action request may be tostart or to end an application or to perform a particular action withinthe application. Input/output interface 140 may include one or moreinput devices. Example input devices may include a keyboard, a mouse, agame controller, a glove, a button, a touch screen, or any othersuitable device for receiving action requests and communicating thereceived action requests to console 110. An action request received bythe input/output interface 140 may be communicated to console 110, whichmay perform an action corresponding to the requested action. In someembodiments, input/output interface 140 may provide haptic feedback tothe user in accordance with instructions received from console 110. Forexample, input/output interface 140 may provide haptic feedback when anaction request is received, or when console 110 has performed arequested action and communicates instructions to input/output interface140.

Console 110 may provide content to near-eye display 120 for presentationto the user in accordance with information received from one or more ofexternal imaging device 150, near-eye display 120, and input/outputinterface 140. In the example shown in FIG. 1, console 110 may includean application store 112, a headset tracking module 114, a virtualreality engine 116, and eye-tracking module 118. Some embodiments ofconsole 110 may include different or additional modules than thosedescribed in conjunction with FIG. 1. Functions further described belowmay be distributed among components of console 110 in a different mannerthan is described here.

In some embodiments, console 110 may include a processor and anon-transitory computer-readable storage medium storing instructionsexecutable by the processor. The processor may include multipleprocessing units executing instructions in parallel. Thecomputer-readable storage medium may be any memory, such as a hard diskdrive, a removable memory, or a solid-state drive (e.g., flash memory ordynamic random access memory (DRAM)). In various embodiments, themodules of console 110 described in conjunction with FIG. 1 may beencoded as instructions in the non-transitory computer-readable storagemedium that, when executed by the processor, cause the processor toperform the functions further described below.

Application store 112 may store one or more applications for executionby console 110. An application may include a group of instructions that,when executed by a processor, generates content for presentation to theuser. Content generated by an application may be in response to inputsreceived from the user via movement of the user's eyes or inputsreceived from the input/output interface 140. Examples of theapplications may include gaming applications, conferencing applications,video playback application, or other suitable applications.

Headset tracking module 114 may track movements of near-eye display 120using slow calibration information from external imaging device 150. Forexample, headset tracking module 114 may determine positions of areference point of near-eye display 120 using observed locators from theslow calibration information and a model of near-eye display 120.Headset tracking module 114 may also determine positions of a referencepoint of near-eye display 120 using position information from the fastcalibration information. Additionally, in some embodiments, headsettracking module 114 may use portions of the fast calibrationinformation, the slow calibration information, or some combinationthereof, to predict a future location of near-eye display 120. Headsettracking module 114 may provide the estimated or predicted futureposition of near-eye display 120 to VR engine 116.

Headset tracking module 114 may calibrate the artificial reality systemenvironment 100 using one or more calibration parameters, and may adjustone or more calibration parameters to reduce errors in determining theposition of near-eye display 120. For example, headset tracking module114 may adjust the focus of external imaging device 150 to obtain a moreaccurate position for observed locators on near-eye display 120.Moreover, calibration performed by headset tracking module 114 may alsoaccount for information received from IMU 132. Additionally, if trackingof near-eye display 120 is lost (e.g., external imaging device 150 losesline of sight of at least a threshold number of locators 126), headsettracking module 114 may re-calibrate some or all of the calibrationparameters.

VR engine 116 may execute applications within artificial reality systemenvironment 100 and receive position information of near-eye display120, acceleration information of near-eye display 120, velocityinformation of near-eye display 120, predicted future positions ofnear-eye display 120, or some combination thereof from headset trackingmodule 114. VR engine 116 may also receive estimated eye position andorientation information from eye-tracking module 118. Based on thereceived information, VR engine 116 may determine content to provide tonear-eye display 120 for presentation to the user. For example, if thereceived information indicates that the user has looked to the left, VRengine 116 may generate content for near-eye display 120 that mirrorsthe user's eye movement in a virtual environment. Additionally, VRengine 116 may perform an action within an application executing onconsole 110 in response to an action request received from input/outputinterface 140, and provide feedback to the user indicating that theaction has been performed. The feedback may be visual or audiblefeedback via near-eye display 120 or haptic feedback via input/outputinterface 140.

Eye-tracking module 118 may receive eye-tracking data from eye-trackingunit 130 and determine the position of the user's eye based on the eyetracking data. The position of the eye may include an eye's orientation,location, or both relative to near-eye display 120 or any elementthereof. Because the eye's axes of rotation change as a function of theeye's location in its socket, determining the eye's location in itssocket may allow eye-tracking module 118 to more accurately determinethe eye's orientation.

In some embodiments, eye-tracking unit 130 may output eye-tracking dataincluding images of the eye, and eye-tracking module 118 may determinethe eye's position based on the images. For example, eye-tracking module118 may store a mapping between images captured by eye-tracking unit 130and eye positions to determine a reference eye position from an imagecaptured by eye-tracking unit 130. Alternatively or additionally,eye-tracking module 118 may determine an updated eye position relativeto a reference eye position by comparing an image from which thereference eye position is determined to an image from which the updatedeye position is to be determined. Eye-tracking module 118 may determineeye position using measurements from different imaging devices or othersensors. For example, as described above, eye-tracking module 118 mayuse measurements from a slow eye-tracking system to determine areference eye position, and then determine updated positions relative tothe reference eye position from a fast eye-tracking system until a nextreference eye position is determined based on measurements from the sloweye-tracking system.

Eye-tracking module 118 may also determine eye calibration parameters toimprove precision and accuracy of eye tracking. Eye calibrationparameters may include parameters that may change whenever a user donsor adjusts near-eye display 120. Example eye calibration parameters mayinclude an estimated distance between a component of eye-tracking unit130 and one or more parts of the eye, such as the eye's center, pupil,cornea boundary, or a point on the surface of the eye. Other example eyecalibration parameters may be specific to a particular user and mayinclude an estimated average eye radius, an average corneal radius, anaverage sclera radius, a map of features on the eye surface, and anestimated eye surface contour. In embodiments where light from theoutside of near-eye display 120 may reach the eye (as in some augmentedreality applications), the calibration parameters may include correctionfactors for intensity and color balance due to variations in light fromthe outside of near-eye display 120. Eye-tracking module 118 may use eyecalibration parameters to determine whether the measurements captured byeye-tracking unit 130 would allow eye-tracking module 118 to determinean accurate eye position (also referred to herein as “validmeasurements”). Invalid measurements, from which eye-tracking module 118may not be able to determine an accurate eye position, may be caused bythe user blinking, adjusting the headset, or removing the headset,and/or may be caused by near-eye display 120 experiencing greater than athreshold change in illumination due to external light.

FIG. 2 is a perspective view of a simplified example near-eye display200 including various sensors. Near-eye display 200 may be a specificimplementation of near-eye display 120 of FIG. 1, and may be configuredto operate as a virtual reality display, an augmented reality display,and/or a mixed reality display. Near-eye display 200 may include a frame205 and a display 210. Display 210 may be configured to present contentto a user. In some embodiments, display 210 may include displayelectronics and/or display optics. For example, as described above withrespect to near-eye display 120 of FIG. 1, display 210 may include anLCD display panel, an LED display panel, or an optical display panel(e.g., a waveguide display assembly).

Near-eye display 200 may further include various sensors 250 a, 250 b,250 c, 250 d, and 250 e on or within frame 205. In some embodiments,sensors 250 a-250 e may include one or more depth sensors, motionsensors, position sensors, inertial sensors, or ambient light sensors.In some embodiments, sensors 250 a-250 e may include one or more imagesensors configured to generate image data representing different fieldsof views in different directions. In some embodiments, sensors 250 a-250e may be used as input devices to control or influence the displayedcontent of near-eye display 200, and/or to provide an interactiveVR/AR/MR experience to a user of near-eye display 200. In someembodiments, sensors 250 a-250 e may also be used for stereoscopicimaging.

In some embodiments, near-eye display 200 may further include one ormore illuminators 230 to project light into the physical environment.The projected light may be associated with different frequency bands(e.g., visible light, infra-red light, ultra-violet light, etc.), andmay serve various purposes. For example, illuminator(s) 230 may projectlight in a dark environment (or in an environment with low intensity ofinfra-red light, ultra-violet light, etc.) to assist sensors 250 a-250 ein capturing images of different objects within the dark environment. Insome embodiments, illuminator(s) 230 may be used to project certainlight pattern onto the objects within the environment. In someembodiments, illuminator(s) 230 may be used as locators, such aslocators 126 described above with respect to FIG. 1.

In some embodiments, near-eye display 200 may also include ahigh-resolution camera 240. Camera 240 may capture images of thephysical environment in the field of view. The captured images may beprocessed, for example, by a virtual reality engine (e.g., virtualreality engine 116 of FIG. 1) to add virtual objects to the capturedimages or modify physical objects in the captured images, and theprocessed images may be displayed to the user by display 210 for AR orMR applications.

FIG. 3 is a perspective view of a simplified example near-eye display300 including an example eye-tracking unit. FIG. 3 may be theperspective view of near-eye display 300 viewed from the side that facesthe eyes of the user. As near-eye display 200, near-eye display 300 mayinclude a frame 305 and a display 310. Frame 305 may be coupled to orembedded with one or more electrical or optical components. Display 310may include display electronics and/or display optics, and may beconfigured to present content to a user. For example, as describedabove, display 310 may include an LCD display panel, an LED displaypanel, and/or an optical display panel (e.g., a waveguide displayassembly).

Near-eye display 300 may include one or more light sources 320 and oneor more cameras 330. As discussed in further detail below, lightsource(s) 320 may be mounted on a substrate 340, such that lightsource(s) 320 are positioned within the field of view of the eye of theuser. Any suitable number of light source(s) 320 may be used, and lightsource(s) 320 may be arranged in any suitable pattern, such as aone-dimensional array or a two-dimensional array. Light source(s) 320may be spaced closer together or farther apart than shown in FIG. 3.Substrate 340 may be mounted in front of the display 310, or may beintegrated with the display 310. Substrate 340 may be transparent tovisible light. Camera(s) 330 may be coupled to or embedded in frame 305.Light source(s) 320 may emit light in certain frequency range (e.g.,NIR) towards the eye of the user. The emitted light may be reflected bythe eyes of the user. The reflected light may then be received bycamera(s) 330 to form images that may indicate certain characteristicsof light source(s) 320 and the eyes of the user. Based on the imagescaptured by camera(s) 330, an eye's position, including the orientationand location of the eye, may be determined. The gaze direction and/orgaze point of the user may also be determined based on the detectedeye's position as described above with respect to FIG. 1. The imagecontent displayed on display 310 may then be adjusted accordingly basedon the gaze direction and/or gaze point of the user.

FIG. 4 is a cross-sectional view of an example near-eye display 400including an example eye-tracking unit. It is noted that, even thoughFIG. 4 and other figures in the present disclosure show an eye of a userof a near-eye display for illustration purposes, the eye of the user isnot a part of the corresponding near-eye display. Like near-eye displays200 and 300, near-eye display 400 may include a frame 405 and a displaysystem that includes display electronics 410 and/or display optics 420coupled to or embedded in frame 405. As described above with respect todisplay electronics 122, display electronics 410 may display images tothe user according to data received from a console, such as console 110.Display electronics 410 may include sub-pixels to emit light of apredominant color, such as red, green, blue, white, or yellow. Displayoptics 420 may display image content optically (e.g., using opticalwaveguides and optical couplers), or magnify image light emitted bydisplay electronics 410, correct optical errors associated with theimage light, and present the corrected image light to the user ofnear-eye display 400. In various embodiments, display optics 420 mayinclude one or more optical elements. Example optical elements mayinclude a substrate, optical waveguides, optical couplers, an aperture,a Fresnel lens, a convex lens, a concave lens, a filter, or any othersuitable optical element that may affect image light emitted fromdisplay electronics 410. Display optics 420 may include a combination ofdifferent optical elements as well as mechanical couplings to maintainrelative spacing and orientation of the optical elements in thecombination. One or more optical elements in display optics 420 may havean optical coating, such as an anti-reflective coating, a reflectivecoating, a filtering coating, or a combination of different opticalcoatings.

Near-eye display 400 may include an eye-tracking unit that includes alight source 430 and a camera 440. Light source 430 may be mounted onsubstrate 422, which may be mounted on, coupled to, or embedded in frame450 in front of display optics 420. Substrate 450 may be transparent tovisible light. Camera 440 may be mounted on, coupled to, or embedded inframe 405. Light source 430 may emit light towards an eye 490 of theuser of near-eye display 400, and may be positioned within a field ofview of eye 490. The emitted light may be reflected by the cornea 492 ofeye 490 of the user. The reflected light may then be further reflectedby reflector 424-1 and received by camera 440 to generate images thatmay indicate certain characteristics of light source 430 and eye 490 ofthe user. Based on the images captured by camera 440, the position ofeye 490, including the orientation and location of eye 490, may bedetermined. The gaze direction and/or gaze point of the user may bedetermined based on the detected position of eye 490 as described abovewith respect to FIG. 1. The image content displayed on the displaysystem may then be adjusted accordingly based on the gaze directionand/or gaze point of the user. Reflector 424-2 may be used to shadowlight source 430 from the field of view of camera 440, as discussed infurther detail below.

In some implementations, light source 430 may include a coherent lightsource (i.e., a light source emitting light at a precise wavelength withnegligible phase difference), such as a VCSEL. The VCSEL may illuminatea portion of the surface of eye 490, such as cornea 492 or iris 494,with coherent light. For example, the VCSEL may emit light in theinfrared spectrum having a wavelength between about 830 nm and about 860nm. As another example, the VCSEL may emit light having a wavelengthbetween about 900 nm and about 1160 nm, such as between about 930 nm andabout 980 nm. Alternatively, the VCSEL may emit light having awavelength in the visible spectrum. However, illuminating the eye withlight in the infrared spectrum may reduce interference and noise fromvisible light emitted by display electronics 410 or from externalvisible light that passes into near-eye display 400, as in someaugmented reality applications. The VCSEL may have a low power toprevent user discomfort or injury.

Although light source 430 may typically include a coherent light source,non-coherent light sources may be used in some implementations. Forexample, in some implementations, light source 430 may include an LEDemitting light with wavelengths in the visible band or in the infraredband. For example, light source 430 may include a micro-LED. However,because LEDs emit light across a broader wavelength range than a laser,LEDs may produce images with lower contrast than those produced using acoherent light source. In some embodiments, an additional light sourcethat emits light at a different wavelength than the light source may beused to increase eye-tracking precision. Although a single light source430 and two reflectors 424-1 and 424-2 are shown in FIG. 4, any suitablenumber of light sources and reflectors may be used.

Camera 440 may capture light reflected by the portion of the eye surfaceilluminated by light source 430 and further reflected by reflector424-1. In one example, camera 440 may capture an image with a pixelarray of 30 by 30 pixels, where a pixel may correspond to a resolutionof about 15 to 40 μm of the eye surface. In this example, the imagedportion of the surface of eye 490 may have an area of between about 0.20and about 1.44 square millimeters. In various embodiments, camera 440may have increased resolution to increase eye tracking precision andaccuracy. For example, camera 440 may have a quarter video graphic array(QVGA) resolution with a pixel array of 320×240 pixels. Increasing thenumber of pixels included in camera 440 may allow the size of thesurface of eye 490 corresponding to a pixel to be decreased, allow thearea of the surface of eye 490 imaged by camera 440 to be increased, orboth. However, using fewer pixels may beneficially reduce the powerconsumption of camera 440, and illuminating and imaging a smaller areaof the surface of the eye may beneficially reduce power consumption bylight source 430. In some embodiments, camera 440 may include an opticalmouse sensor or other sensor capturing images at a very high frame rate.For example, in some cases, camera 440 may capture about 5,000 imagesper second to provide precise eye tracking data. Although a singlecamera 440 is shown in FIG. 4, any suitable number of cameras may beused.

FIG. 5 illustrates light reflections and diffusions by an eye 550 duringeye tracking using an eye-tracking unit 510. Eye-tracking unit 510 mayinclude a light source 512, a reflector 534-1, and a reflector 534-2 asdescribed above with respect to FIG. 4. Although only one light source512 is shown in FIG. 5, it should be understood that multiple additionallight sources may be placed within the field of view of eye 550. Inother embodiments, eye-tracking unit 510 may include different and/oradditional components than those depicted in FIG. 4. Light source 512may include a VCSEL or a micro-LED and may emit light having a centeraxis that forms an angle 522 relative to a surface normal vector 520 ofeye 550. As discussed in further detail below, a beam divertingcomponent may be positioned proximate to an emission area of lightsource 512 in order to achieve a desired light output direction. Surfacenormal vector 520 is orthogonal to a portion of the surface (e.g.,cornea 552) of eye 550 illuminated by light source 512. In the exampleshown in FIG. 5, surface normal vector 520 may be the same as the fovealaxis (a line from the center of pupil 556 to fovea 562) of eye 550.Alternatively, surface normal vector 520 may be orthogonal to anotherportion of the surface of eye 550. Angle 522 may be measured betweensurface normal vector 520 and a line from a center of the portion of thesurface of eye 550 illuminated by light source 512 to a center of theoutput aperture of light source 512. Angle 522 may be chosen to have anysuitable value, such that at least a portion of the light from lightsource 512 is reflected by eye 550 and received by reflector 534-1. Forexample, angle 522 may be chosen to be close to zero (e.g., between 5°and 10°) in order to minimize any distortions caused by larger incidentangles, but greater than zero to prevent the light from being reflecteddirectly back at light source 512 by eye 550. Various reflectors and/orother optical components, such as reflector 534-1, may be used to directlight reflected by eye 550 toward a camera 540. For example, a pluralityof reflectors may be immersed in a transparent substrate that may bepositioned within the field of view of eye 550.

Reflector 534-1 may be mounted at an angle 524 relative to surfacenormal vector 520 of eye 550. Reflector 534-1 may be mounted within thefield of view of eye 550. Angle 524 may be measured between surfacenormal vector 520 and a line from a center of the portion of the surfaceof eye 550 illuminated by light source 512 to a center of reflector534-1. In some embodiments, a difference between angle 522 and angle 524may be less than a threshold amount so that reflector 534-1 may reflectimages via specular reflections of light incident on cornea 552 of eye550, which may beneficially increase contrast of the resulting image andminimize light power loss and power consumption.

The light emitted by light source 512 may substantially uniformlyilluminate a portion of the eye surface (e.g., cornea 552). A portion ofthe emitted light may be reflected specularly by cornea 552 of eye 550and incident on reflector 534-1. In some cases, the light incident oneye 550 may propagate into the eye for a small distance before beingreflected. At least some portions of the light may enter eye 550 throughcornea 552 and reach iris 554, pupil 556, lens 558, or retina 560 of eye550. Because the eye surface and the interfaces within eye 550 (e.g.,surface of iris 554 or pupil 556) may be rough (e.g., due to featuressuch as capillaries or bumps), the eye surface and the interfaces withineye 550 may scatter the incident light in multiple directions. Differentportions of the eye surface and the interfaces within eye 550 may havedifferent arrangements of features. Thus, an intensity pattern of thelight reflected by eye 550 may depend on the arrangement of featureswithin the illuminated portion of eye 550, which may allowidentification of the portion of the eye from the intensity pattern.

Camera 540 may collect and project light reflected by reflector 534-1onto an image sensor of camera 540. Camera 540 may also correct one ormore optical errors (such as those described with respect to the displayoptics 124) to improve the contrast and other properties of the imagescaptured by the image sensor of camera 540. In some embodiments, camera540 may also magnify the reflected light. In some embodiments, camera540 may enlarge the images. The image sensor of camera 540 may captureincident light focused by a lens assembly of camera 540. Thus, camera540 may effectively capture an image of light source 512 (the emittedlight of which is reflected specularly by the cornea of the eye)reflected by the eye, resulting in a “glint” in the captured image.Because of the scattering (diffusive reflections) at the eye surface andinternal interfaces of the eye, light incident on a point of the imagesensor may include light reflected from multiple points within theilluminated portion of eye 550, and thus may be the result of theinterference of the light reflected from the multiple points. Thus, theimage sensor of camera 540 may also capture a diffraction or specklepattern formed by a combination of light reflected from multiple pointsof the surface of eye 550.

Each pixel of the image sensor may include a light-sensitive circuitthat can output a current or voltage signal proportional to theintensity of the light incident on the pixel. In some embodiments, thepixels of the image sensor may be sensitive to light in a narrowwavelength band. In some other embodiments, the pixels of the imagesensor may have a wide-band or multi-band sensitivity. For example, theimage sensor of camera 540 may include a complementary metal-oxidesemiconductor (CMOS) pixel array, which may be used with light having awavelength less than about 850 nm. As another example, the image sensorof camera 514 may include an indium gallium arsenide (InGaAs) alloypixel array. Such an image sensor may be used with light having awavelength between about 900 nm and about 1160 nm.

In some embodiments, to determine a position change of eye 550, aneye-tracking module (e.g., eye-tracking module 118 of FIG. 1) maydetermine a pixel shift between images. Multiplying the pixel shift by acalibrated distance per pixel may allow the eye-tracking module todetermine a distance the surface (e.g., cornea 552) of eye 550 hasshifted. For example, if the glint captured in one image is shifted bytwo pixels relative to the glint captured in a previous image, and eachpixel corresponds to a distance of 10 μm at the surface of eye 550, thesurface of eye 550 may have moved about 20 μm.

Alternatively or additionally, the eye-tracking module may determine theposition of the eye in a captured image by comparing the captured imageswith one or more previous images having known positions of the eye. Forexample, the eye-tracking module may include a database of images thatare each associated with a reference eye position. By matching thecaptured image with a stored image, the eye-tracking module maydetermine that the eye is at the reference eye position associated withthe stored image. In some embodiments, the eye-tracking module mayidentify a feature in a portion of a captured image. The feature mayinclude a diffraction or optical flow pattern associated with aparticular portion of eye 550. For example, the eye-tracking module maydetermine the eye position by retrieving a reference eye positionassociated with the feature (which was also captured in a referenceimage), determining a pixel shift between the feature in the capturedimage and the feature in the reference image, and determining the eyeposition by modifying the reference eye position according to thedetermined pixel shift using the calibrated distance per pixel asdescribed above.

As discussed above, camera 540 may effectively capture an image of lightsource 512 reflected by cornea 552 of eye 550. In some cases, the lightsource may be an extended source rather than a point source. Thus, thecaptured image (i.e., glint) of light source 512 may have a shape of acircle, a rectangle, an oval, or an irregular shape, and the spatialstructure of light source 512 may be captured in the image. The extendedshape of the glint and/or the spatial structure captured in the image ofthe light source may cause errors when determining the relative locationof the glint in the image using, for example, the centroiding algorithm.The errors may affect the accuracy of eye tracking when the relativelocation (e.g., pixel shift) of the glint in the image is used todetermine the corneal location in 3D space. Therefore, the light source512 may have an emission area with a small form factor that is much lessthan 200 μm. The small emission area of a VCSEL or a micro-LED wouldappear more like a point source than an extended source in the image andreduce the size of the resulting glint on the captured image. A VCSEL ora micro-LED that has a bare die size less than 200 μm would have anemission area with a linear dimension that is significantly smaller than200 μm because the emission area of the VCSEL or micro-LED is muchsmaller than the bare die size in order to accommodate other components,such as the bond part. For example, a diameter of the emission area maybe less than 20 μm. A smaller glint size in the captured image may leadto a more precise glint location determination and more accurate eyetracking.

According to certain aspects of the present disclosure, a light sourceor a plurality of light sources may be mounted on a transparentsubstrate that can be positioned in front of the user, such that thelight source or plurality of light sources are within the user's fieldof view. The light source or plurality of light sources may be used forillumination and imaging in eye tracking. A beam diverting component maybe provided for each light source, in order to direct light from thelight source toward the eye of the user. The beam diverting componentmay cause at least a portion of the light from the light source to beincident on the eye of the user at an angle with respect to a vectorthat is normal to a surface of the eye of the user.

Further, according to certain aspects of the present disclosure, aplurality of reflectors (e.g., dichroic mirrors) may be immersed in thetransparent substrate that can be positioned in front of the user andwithin the user's field of view. As discussed in further detail below,the reflectors may shadow the light sources, such that light from thelight sources does not directly reach the camera. Each reflector mayreflect light in a first band of the optical spectrum (e.g., IR light orMR light) and transmit light in a second band of the optical spectrum(e.g., visible light). The reflectors may be have a flat, spherical,aspherical, anamorphic, or cylindrical surface. The reflectors may havea reflectivity of at least 30%, at least 50%, and at least 70%, or more,in the desired band of the optical spectrum. The (photopically weighted)transmissivity of the reflector in the working wavelength range (e.g.,visible light) may be at least 80%, at least 90%, at least 95%, at least99%, or higher. Suitable reflectors may include multilayer dielectricreflectors, metallic coatings, and transparent conductive films. Thesubstrate may be transparent to both light in the first band and lightin the second band.

In some implementations, the reflectors may be hot mirrors that canreflect IR light but are transparent to visible light. The IR lightreflectivity of and visible light transmissivity of the reflectors mayvary in a same near-eye display device or in different same near-eyedisplay devices. Because visible light is allowed to pass through thereflectors and the substrate, the reflectors can be positioned in frontof the user's eyes without obstructing the user's field of view. Forexample, in an AR application, the user can look through the substrateand the immersed reflectors to see the outside world. In a VRapplication, the user can look through the substrate and the immersedreflectors to view the displayed content. At the same time, light from alight source for eye illumination may be reflected by the user's eye(e.g., cornea), and may then be reflected by the reflectors to a camerato form the glints in images captured by the camera for eye tracking.

FIG. 6 is a simplified diagram of an example system 600 for eye trackingin an example near-eye display, according to certain embodiments. FIG. 6is merely illustrative and is not drawn to scale. System 600 may includea light source 612 that may be mounted on a mounting surface 628 of aprism 604. Light source 612 may be mounted on mounting surface 628 inany suitable manner, such as die bonding. Light source 612 may bepositioned within a field of view of an eye of a user. As discussedabove, light source 612 may have a very small form factor, such as lessthan 200 μm, such that light source 612 is effectively invisible to theuser. Light source 612 may have an emission cone 630 with an angle thatis less than 30°.

System 600 may also include a prism 602 on which a reflective coating622 is formed. Reflective coating 622 may be configured to shadow lightsource 612 from a field of view of camera 650. Shadowing light fromlight source 612 and preventing it from directly reaching camera 650 isadvantageous for contrast, image quality, and the signal-to-noise ratiofor the image processing phase of eye-tracking. Without shadowing lightfrom light source 612, camera 650 may be subject to unnecessary straylight.

Reflective coating 622 may include multiple layers of dielectricmaterials that are formed on a surface of prism 602. Reflective coating622 may be formed by a directional coating process, such as electronbeam deposition, ion beam deposition, or atomic layer deposition. Asource for the deposition material may be positioned at the location ofcamera 650. During deposition, prism 602 shadows a portion of prism 604,such that an uncoated surface 626 of prism 604 remains, while areflective coating 624 is deposited on an unshadowed portion of prism604. Light source 612 may be positioned such that emitted light travelsthrough uncoated surface 626 of prism 604. Emission cone 630 may benarrow enough that emitted light propagates between reflective coatings622 and 624, and is not incident on reflective coatings 622 or 624,thereby preventing stray light from being reflected toward mountingsurface 628. Further, reflective coating 622 prevents light from lightsource 612 from reaching camera 650 directly, thereby preventing noisein the captured images of the eye of the user and misidentification offeatures of the eye of the user.

Light source 612 may be configured to emit light toward the eye of theuser. The light may then be reflected by the eye of the user towardreflective coating 624, which further reflects the light toward camera650. Reflective coating 624 may be tilted at an angle such that thereflected light reaches camera 650. If multiple light sources andmultiple corresponding reflective coatings are used, each reflectivecoating may be tilted at a different angle to reflect light from the eyeof the user toward camera 650.

Reflective coatings 622 and 624 may be hot mirror coatings that reflectnear infrared light and transmit visible light. Further, reflectivecoatings 622 and 624 may be formed on a flat surface of prisms 602 and604, respectively. A Fresnel lens may be formed on each flat surface inorder to focus the reflected light. Alternatively, reflective coatings622 and 624 may be formed on a curved surface of prisms 602 and 604,respectively. The curved surface may include a concave surface, a convexsurface, a cylindrical surface, an aspherical surface, and/or a freeformsurface.

FIG. 7 is a simplified diagram of an example illumination system 700 foreye tracking in an example near-eye display, according to certainembodiments. FIG. 7 is merely illustrative and is not drawn to scale.Illumination system 700 may include a light source 712 that may bemounted on a mounting surface 728 of a substrate 716. Light source 712may be mounted on mounting surface 728 in any suitable manner, such asdie bonding. Light source 712 may be positioned within a field of viewof an eye of a user. As discussed above, light source 712 may have avery small form factor, such as less than 200 μm, such that light source712 is effectively invisible to the user. Substrate 716 may be made ofany suitable material that is transparent to visible and near infraredlight, such as glass, quartz, plastic, polymer, ceramic, or crystal.

System 700 may also include reflectors 722, 724, and 726. Reflector 722may be configured to shadow light source 712 from a field of view of acamera (not shown). Reflectors 722, 724, and 726 may be formed in asimilar manner as described above with regard to FIG. 6. Alternatively,reflectors 722, 724, and 726 may be formed in any suitable manner, andmay be standalone mirrors, dielectric stacks, reflective coatings, orany other reflective material. Reflectors 722, 724, and 726 may beFresnel reflectors.

Light source 712 may be positioned such that emitted light travelsbetween reflectors 722 and 724, and is not incident on reflectivecoatings 722 or 724, thereby preventing stray light from being reflectedtoward mounting surface 728. Further, reflector 722 prevents light fromlight source 712 from reaching the camera directly, thereby preventingnoise in the captured images of the eye of the user.

Light source 712 may be configured to emit light toward the eye of theuser. The emitted light 738 may propagate through an outcoupling surface732, and may then be reflected by the eye of the user toward reflector724, which further reflects the light toward the camera. Reflector 724may be tilted at an angle such that the reflected light reaches thecamera. If multiple light sources and multiple corresponding reflectivecoatings are used, each reflective coating may be tilted at a differentangle to reflect light from the eye of the user toward the camera.

Reflectors 722 and 724 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 722 and724 may be formed on a flat surface of prisms within substrate 716. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 722 and 724 may be formed ona curved surface of prisms within substrate 716. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface. In the example shown inFIG. 7, reflector 724 is formed by coating approximately 80% of thesurface of a prism with a hot mirror coating.

FIGS. 8A-8C are simplified diagrams of example illumination systems 800,801, and 802 for eye tracking in example near-eye displays, according tocertain embodiments. FIGS. 8A-8C are merely illustrative and are notdrawn to scale. FIGS. 8A-8C illustrate the use of beam divertingcomponents to direct light from a light source toward the eye of theuser. In the examples shown in FIGS. 8A-8C, the beam divertingcomponents may be indented prisms or cones that are formed at anoutcoupling surface of a substrate. Alternatively or in addition, thebeam diverting components may be diffraction gratings or lenses that areformed at the outcoupling surface of the substrate.

As shown in FIG. 8A, illumination system 800 may include a light source812 that may be mounted on a mounting surface 828 of a substrate 816.Light source 812 may be mounted on mounting surface 828 in any suitablemanner, such as die bonding. Light source 812 may be positioned within afield of view of an eye of a user. As discussed above, light source 812may have a very small form factor, such as less than 200 μm, such thatlight source 812 is effectively invisible to the user. Substrate 816 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 800 may also include reflectors 822 and 824.Reflector 822 may be configured to shadow light source 812 from a fieldof view of a camera (not shown). Reflectors 822 and 824 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 822 and 824 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 822 and 824 maybe Fresnel reflectors.

Light source 812 may be positioned such that a first portion of light830 travels between reflectors 822 and 824, a second portion of light830 is transmitted by reflector 822 toward outcoupling surface 832, anda third portion of light 830 is reflected by reflector 822 towardreflector 824. Some of the third portion of light 830 passes throughreflector 824 and is transmitted through mounting surface 828 as straylight 834. The rest of the third portion of light 830 is reflected byreflector 824 toward outcoupling surface 832. Reflector 822 may preventlight from light source 812 from reaching the camera directly, therebypreventing noise in the captured images of the eye of the user.

The spacing between reflectors 822 and 824 may be adjusted to optimizeimage quality. For optimal imaging conditions, reflectors 822 and 824may be as close to each other as possible. However, if reflectors 822and 824 are too close to each other, some of the light may be lost dueto multi-path reflections from reflectors 822 and 824. Also, thisconfiguration may lead to stray light 834, as shown in FIG. 8A. On theother hand, for optimal illumination conditions, reflectors 822 and 824may be as far from each other as possible. However, if reflectors 822and 824 are too far apart, there may be a loss of signal in the imagingpath for the light reflected by the eye and incident on the camera.Accordingly, a compromise may be reached that optimizes the overallquality of the image produced by the camera.

Indented prism 836 may be used as a beam diverting component to redirectlight from light source 812 toward the eye of the user. Indented prism836 may be formed at outcoupling surface 832 of substrate 816 by anysuitable method, such as gray-scale lithography. A shape of indentedprism 836 may be configured to provide an amount of tilt such thatemitted light 838 reaches the eye of the user.

The emitted light 838 may propagate through outcoupling surface 832, andmay then be reflected by the eye of the user toward reflector 824, whichfurther reflects the light toward the camera. Reflector 824 may betilted at an angle such that the reflected light reaches the camera. Ifmultiple light sources and multiple corresponding reflective coatingsare used, each reflective coating may be tilted at a different angle toreflect light from the eye of the user toward the camera.

Reflectors 822 and 824 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 822 and824 may be formed on a flat surface of prisms within substrate 816. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 822 and 824 may be formed ona curved surface of prisms within substrate 816. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

As shown in FIG. 8B, illumination system 801 may include a light source842 that may be mounted on a mounting surface 858 of a substrate 846.Light source 842 may be mounted on mounting surface 858 in any suitablemanner, such as die bonding. Light source 842 may be positioned within afield of view of an eye of a user. As discussed above, light source 842may have a very small form factor, such as less than 200 μm, such thatlight source 842 is effectively invisible to the user. Substrate 846 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 801 may also include reflectors 852 and 854.Reflector 852 may be configured to shadow light source 842 from a fieldof view of a camera (not shown). Reflectors 852 and 854 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 852 and 854 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 852 and 854 maybe Fresnel reflectors.

Light source 842 may be positioned such that a first portion of light860 travels between reflectors 852 and 854, a second portion of light860 is transmitted by reflector 852 toward outcoupling surface 862, anda third portion of light 860 is reflected by reflector 852 towardreflector 854. Some of the third portion of light 860 passes throughreflector 854 and is transmitted through mounting surface 858 as straylight 864. The rest of the third portion of light 860 is reflected byreflector 854 toward outcoupling surface 852. Reflector 852 may preventlight from light source 842 from reaching the camera directly, therebypreventing noise in the captured images of the eye of the user.

The spacing between reflectors 852 and 854 may be adjusted to optimizeimage quality. For optimal imaging conditions, reflectors 852 and 854may be as close to each other as possible. However, if reflectors 852and 854 are too close to each other, some of the light may be lost dueto multi-path reflections from reflectors 852 and 854. Also, thisconfiguration may lead to stray light 864, as shown in FIG. 8B. On theother hand, for optimal illumination conditions, reflectors 852 and 854may be as far from each other as possible. However, if reflectors 852and 854 are too far apart, there may be a loss of signal in the imagingpath for the light reflected by the eye and incident on the camera.Accordingly, a compromise may be reached that optimizes the overallquality of the image produced by the camera.

Indented prism 866 may be used as a beam diverting component to redirectlight from light source 842 toward the eye of the user. Indented prism866 may be formed at outcoupling surface 862 of substrate 846 by anysuitable method, such as gray-scale lithography. A shape of indentedprism 866 may be configured to provide an amount of tilt such thatemitted light 868 reaches the eye of the user. As shown in FIG. 8B,indented prism 866 provides a greater amount of tilt than indented prism836.

The emitted light 868 may propagate through outcoupling surface 862, andmay then be reflected by the eye of the user toward reflector 854, whichfurther reflects the light toward the camera. Reflector 854 may betilted at an angle such that the reflected light reaches the camera. Ifmultiple light sources and multiple corresponding reflective coatingsare used, each reflective coating may be tilted at a different angle toreflect light from the eye of the user toward the camera.

Reflectors 852 and 854 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 852 and854 may be formed on a flat surface of prisms within substrate 846. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 852 and 854 may be formed ona curved surface of prisms within substrate 846. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

As shown in FIG. 8C, illumination system 802 may include a light source872 that may be mounted on a mounting surface 888 of a substrate 876.Light source 872 may be mounted on mounting surface 888 in any suitablemanner, such as die bonding. Light source 872 may be positioned within afield of view of an eye of a user. As discussed above, light source 872may have a very small form factor, such as less than 200 μm, such thatlight source 872 is effectively invisible to the user. Substrate 876 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 802 may also include reflectors 882 and 884.Reflector 882 may be configured to shadow light source 872 from a fieldof view of a camera (not shown). Reflectors 882 and 884 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 882 and 884 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 882 and 884 maybe Fresnel reflectors.

Light source 872 may be positioned such that a first portion of light890 travels between reflectors 882 and 884, a second portion of light890 is transmitted by reflector 882 toward outcoupling surface 892, anda third portion of light 890 is reflected by reflector 882 towardreflector 884. Some of the third portion of light 890 passes throughreflector 884 and is transmitted through mounting surface 888 as straylight 894. The rest of the third portion of light 890 is reflected byreflector 884 toward outcoupling surface 882. Reflector 882 may preventlight from light source 872 from reaching the camera directly, therebypreventing noise in the captured images of the eye of the user.

The spacing between reflectors 882 and 884 may be adjusted to optimizeimage quality. For optimal imaging conditions, reflectors 882 and 884may be as close to each other as possible. However, if reflectors 882and 884 are too close to each other, some of the light may be lost dueto multi-path reflections from reflectors 882 and 884. Also, thisconfiguration may lead to stray light 894, as shown in FIG. 8C. On theother hand, for optimal illumination conditions, reflectors 882 and 884may be as far from each other as possible. However, if reflectors 882and 884 are too far apart, there may be a loss of signal in the imagingpath for the light reflected by the eye and incident on the camera.Accordingly, a compromise may be reached that optimizes the overallquality of the image produced by the camera.

Indented prism 896 may be used as a beam diverting component to redirectlight from light source 872 toward the eye of the user. Indented prism896 may be formed at outcoupling surface 892 of substrate 876 by anysuitable method, such as gray-scale lithography. A shape of indentedprism 896 may be configured to provide an amount of tilt such thatemitted light 898 reaches the eye of the user. As shown in FIG. 8C,indented prism 896 provides a greater amount of tilt than indentedprisms 836 and 866.

The emitted light 898 may propagate through outcoupling surface 892, andmay then be reflected by the eye of the user toward reflector 884, whichfurther reflects the light toward the camera. Reflector 884 may betilted at an angle such that the reflected light reaches the camera. Ifmultiple light sources and multiple corresponding reflective coatingsare used, each reflective coating may be tilted at a different angle toreflect light from the eye of the user toward the camera.

Reflectors 882 and 884 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 882 and884 may be formed on a flat surface of prisms within substrate 876. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 882 and 884 may be formed ona curved surface of prisms within substrate 876. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

FIGS. 9A-9C are simplified diagrams of example illumination systems 900,901, and 902 for eye tracking in example near-eye displays, according tocertain embodiments. FIGS. 9A-9C are merely illustrative and are notdrawn to scale. FIGS. 9A-9C illustrate the use of beam divertingcomponents to direct light from a light source toward the eye of theuser. In the examples shown in FIGS. 9A-9C, the beam divertingcomponents may be protruding prisms or cones that are formed at anoutcoupling surface of a substrate.

As shown in FIG. 9A, illumination system 900 may include a light source912 that may be mounted on a mounting surface 928 of a substrate 916.Light source 912 may be mounted on mounting surface 928 in any suitablemanner, such as die bonding. Light source 912 may be positioned within afield of view of an eye of a user. As discussed above, light source 912may have a very small form factor, such as less than 200 μm, such thatlight source 912 is effectively invisible to the user. Substrate 916 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 900 may also include reflectors 922 and 924.Reflector 922 may be configured to shadow light source 912 from a fieldof view of a camera (not shown). Reflectors 922 and 924 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 922 and 924 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 922 and 924 maybe Fresnel reflectors.

Light source 912 may be positioned such that light 930 travels betweenreflectors 922 and 924, and is not incident on reflectors 922 or 924,thereby preventing stray light from being reflected toward mountingsurface 928. Further, reflector 922 prevents light from light source 912from reaching the camera directly, thereby preventing noise in thecaptured images of the eye of the user.

Protruding prism 936 may be used as a beam diverting component toredirect light from light source 912 toward the eye of the user.Protruding prism 936 may be formed at outcoupling surface 932 ofsubstrate 916 by any suitable method, such as any suitable depositionmethod. A shape of protruding prism 936 may be configured to provide anamount of tilt such that emitted light 938 reaches the eye of the user.

The emitted light 938 may be reflected by the eye of the user towardreflector 924, which further reflects the light toward the camera.Reflector 924 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera.

Reflectors 922 and 924 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 922 and924 may be formed on a flat surface of prisms within substrate 916. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 922 and 924 may be formed ona curved surface of prisms within substrate 916. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

As shown in FIG. 9B, illumination system 901 may include a light source942 that may be mounted on a mounting surface 958 of a substrate 946.Light source 942 may be mounted on mounting surface 958 in any suitablemanner, such as die bonding. Light source 942 may be positioned within afield of view of an eye of a user. As discussed above, light source 942may have a very small form factor, such as less than 200 μm, such thatlight source 942 is effectively invisible to the user. Substrate 946 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 901 may also include reflectors 952 and 954.Reflector 952 may be configured to shadow light source 942 from a fieldof view of a camera (not shown). Reflectors 952 and 954 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 952 and 954 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 952 and 954 maybe Fresnel reflectors.

Light source 942 may be positioned such that light 960 travels betweenreflectors 952 and 954, and is not incident on reflectors 952 or 954,thereby preventing stray light from being reflected toward mountingsurface 958. Further, reflector 952 prevents light from light source 942from reaching the camera directly, thereby preventing noise in thecaptured images of the eye of the user.

Protruding prism 966 may be used as a beam diverting component toredirect light from light source 942 toward the eye of the user.Protruding prism 966 may be formed at outcoupling surface 962 ofsubstrate 946 by any suitable method, such as any suitable depositionmethod. A shape of protruding prism 966 may be configured to provide anamount of tilt such that emitted light 968 reaches the eye of the user.As shown in FIG. 9B, protruding prism 966 provides a greater amount oftilt than protruding prism 936.

The emitted light 968 may be reflected by the eye of the user towardreflector 954, which further reflects the light toward the camera.Reflector 954 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera.

Reflectors 952 and 954 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 952 and954 may be formed on a flat surface of prisms within substrate 946. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 952 and 954 may be formed ona curved surface of prisms within substrate 946. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

As shown in FIG. 9C, illumination system 902 may include a light source972 that may be mounted on a mounting surface 988 of a substrate 976.Light source 972 may be mounted on mounting surface 988 in any suitablemanner, such as die bonding. Light source 972 may be positioned within afield of view of an eye of a user. As discussed above, light source 972may have a very small form factor, such as less than 200 μm, such thatlight source 972 is effectively invisible to the user. Substrate 976 maybe made of any suitable material that is transparent to visible and nearinfrared light, such as glass, quartz, plastic, polymer, ceramic, orcrystal.

Illumination system 902 may also include reflectors 982 and 984.Reflector 982 may be configured to shadow light source 972 from a fieldof view of a camera (not shown). Reflectors 982 and 984 may be formed ina similar manner as described above with regard to FIG. 6.Alternatively, reflectors 982 and 984 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 982 and 984 maybe Fresnel reflectors.

Light source 972 may be positioned such that light 990 travels betweenreflectors 982 and 984, and is not incident on reflectors 982 or 984,thereby preventing stray light from being reflected toward mountingsurface 988. Further, reflector 982 prevents light from light source 972from reaching the camera directly, thereby preventing noise in thecaptured images of the eye of the user.

Protruding prism 996 may be used as a beam diverting component toredirect light from light source 972 toward the eye of the user.Protruding prism 996 may be formed at outcoupling surface 992 ofsubstrate 976 by any suitable method, such as any suitable depositionmethod. A shape of protruding prism 996 may be configured to provide anamount of tilt such that emitted light 968 reaches the eye of the user.As shown in FIG. 9C, protruding prism 996 provides a greater amount oftilt than protruding prisms 936 and 966.

The emitted light 998 may be reflected by the eye of the user towardreflector 984, which further reflects the light toward the camera.Reflector 984 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera.

Reflectors 982 and 984 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 982 and984 may be formed on a flat surface of prisms within substrate 976. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 982 and 984 may be formed ona curved surface of prisms within substrate 976. The curved surface mayinclude a concave surface, a convex surface, a cylindrical surface, anaspherical surface, and/or a freeform surface.

FIGS. 10A-10C are simplified diagrams of example illumination systems1000, 1001, and 1002 for eye tracking in example near-eye displays,according to certain embodiments. FIGS. 10A-10C are merely illustrativeand are not drawn to scale. FIGS. 10A-10C illustrate the use of beamdiverting components to direct light from a light source toward the eyeof the user. In the examples shown in FIGS. 10A-10C, the beam divertingcomponents may be reflectors that are formed within a substrate.Although FIGS. 10A-10C show that the beam diverting components are madeof two reflectors, the beam diverting components may instead be made ofa single reflector or any suitable number of reflectors. The beamdiverting components may reflect light from a surface that is coated, orby total internal reflection. The beam diverting components may includeany combination of reflectors, prisms, lenses, and/or diffractiongratings.

As shown in FIG. 10A, illumination system 1000 may include a lightsource 1012 that may be mounted on a mounting surface 1028 of asubstrate 1016. Light source 1012 may be mounted on mounting surface1028 in any suitable manner, such as die bonding. Light source 1012 maybe positioned within a field of view of an eye of a user. As discussedabove, light source 1012 may have a very small form factor, such as lessthan 200 μm, such that light source 1012 is effectively invisible to theuser. Substrate 1016 may be made of any suitable material that istransparent to visible and near infrared light, such as glass, quartz,plastic, polymer, ceramic, or crystal.

Illumination system 1000 may also include reflectors 1022 and 1024.Reflector 1022 may be configured to shadow light source 1012 from afield of view of a camera (not shown). Reflectors 1022 and 1024 may beformed in a similar manner as described above with regard to FIG. 6.Alternatively, reflectors 1022 and 1024 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 1022 and 1024 maybe Fresnel reflectors.

Reflectors 1034 and 1036 may be used as beam diverting components toredirect light from light source 1012 toward the eye of the user.Reflectors 1034 and 1036 may be formed in a similar manner as describedabove with regard to FIG. 6. For example, reflectors 1034 and 1036 maybe formed as reflective coatings on prisms, and may be formed at thesame time that reflectors 1022 and 1024 are formed. Alternatively,reflectors 1034 and 1036 may be formed in any suitable manner, and maybe standalone mirrors, dielectric stacks, reflective coatings, or anyother reflective material. For example, reflectors 1034 and 1036 may bedeposited or diamond turned directly onto edge facets of reflectors 1022and 1024, respectively. Reflectors 1034 and 1036 may be angled toprovide an amount of tilt such that emitted light 1038 reaches the eyeof the user.

Light source 1012 may be positioned such that light 1030 is incident onreflector 1034, and is not incident on reflectors 1022 or 1024, therebypreventing stray light from being reflected toward mounting surface1028. Further, reflector 1022 prevents light from light source 1012 fromreaching the camera directly, thereby preventing noise in the capturedimages of the eye of the user. Light 1030 is reflected by reflector 1034toward reflector 1036, and is further reflected by reflector 1036 towardoutcoupling surface 1032.

The emitted light 1038 may be reflected by the eye of the user towardreflector 1024, which further reflects the light toward the camera.Reflector 1024 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera. Although reflector 1036 may be in the field of view of thecamera, reflector 1036 may be small enough that it will be nearlyinvisible to the camera. For example, reflector 1036 may only appear ina single pixel of the focal plane array of the camera. Reflector 1036may have a maximum linear dimension of 100 μm or less.

Reflectors 1022 and 1024 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 1022 and1024 may be formed on a flat surface of prisms within substrate 1016. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 1022 and 1024 may be formedon a curved surface of prisms within substrate 1016. The curved surfacemay include a concave surface, a convex surface, a cylindrical surface,an aspherical surface, and/or a freeform surface.

As shown in FIG. 10B, illumination system 1001 may include a lightsource 1042 that may be mounted on a mounting surface 1058 of asubstrate 1046. Light source 1042 may be mounted on mounting surface1058 in any suitable manner, such as die bonding. Light source 1042 maybe positioned within a field of view of an eye of a user. As discussedabove, light source 1042 may have a very small form factor, such as lessthan 200 μm, such that light source 1042 is effectively invisible to theuser. Substrate 1046 may be made of any suitable material that istransparent to visible and near infrared light, such as glass, quartz,plastic, polymer, ceramic, or crystal.

Illumination system 1001 may also include reflectors 1052 and 1054.Reflector 1052 may be configured to shadow light source 1042 from afield of view of a camera (not shown). Reflectors 1052 and 1054 may beformed in a similar manner as described above with regard to FIG. 6.Alternatively, reflectors 1052 and 1054 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 1052 and 1054 maybe Fresnel reflectors.

Reflectors 1064 and 1066 may be used as beam diverting components toredirect light from light source 1042 toward the eye of the user.Reflectors 1064 and 1066 may be formed in a similar manner as describedabove with regard to FIG. 6. For example, reflectors 1064 and 1066 maybe formed as reflective coatings on prisms, and may be formed at thesame time that reflectors 1052 and 1054 are formed. Alternatively,reflectors 1064 and 1066 may be formed in any suitable manner, and maybe standalone mirrors, dielectric stacks, reflective coatings, or anyother reflective material. For example, reflectors 1064 and 1066 may bedeposited or diamond turned directly onto edge facets of reflectors 1052and 1054, respectively. Reflectors 1064 and 1066 may be angled toprovide an amount of tilt such that emitted light 1068 reaches the eyeof the user. As shown in FIG. 10B, reflector 1066 provides a lesseramount of tilt than reflector 1036.

Light source 1042 may be positioned such that light 1060 is incident onreflector 1064, and is not incident on reflectors 1052 or 1054, therebypreventing stray light from being reflected toward mounting surface1058. Further, reflector 1052 prevents light from light source 1042 fromreaching the camera directly, thereby preventing noise in the capturedimages of the eye of the user. Light 1060 is reflected by reflector 1064toward reflector 1066, and is further reflected by reflector 1066 towardoutcoupling surface 1062.

The emitted light 1068 may be reflected by the eye of the user towardreflector 1054, which further reflects the light toward the camera.Reflector 1054 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera. Although reflector 1066 may be in the field of view of thecamera, reflector 1066 may be small enough that it will be nearlyinvisible to the camera. For example, reflector 1066 may only appear ina single pixel of the focal plane array of the camera. Reflector 1066may have a maximum linear dimension of 100 μm or less.

Reflectors 1052 and 1054 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 1052 and1054 may be formed on a flat surface of prisms within substrate 1046. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 1052 and 1054 may be formedon a curved surface of prisms within substrate 1046. The curved surfacemay include a concave surface, a convex surface, a cylindrical surface,an aspherical surface, and/or a freeform surface.

As shown in FIG. 10C, illumination system 1002 may include a lightsource 1072 that may be mounted on a mounting surface 1088 of asubstrate 1076. Light source 1072 may be mounted on mounting surface1088 in any suitable manner, such as die bonding. Light source 1072 maybe positioned within a field of view of an eye of a user. As discussedabove, light source 1072 may have a very small form factor, such as lessthan 200 μm, such that light source 1072 is effectively invisible to theuser. Substrate 1076 may be made of any suitable material that istransparent to visible and near infrared light, such as glass, quartz,plastic, polymer, ceramic, or crystal.

Illumination system 1002 may also include reflectors 1082 and 1084.Reflector 1082 may be configured to shadow light source 1072 from afield of view of a camera (not shown). Reflectors 1082 and 1084 may beformed in a similar manner as described above with regard to FIG. 6.Alternatively, reflectors 1082 and 1084 may be formed in any suitablemanner, and may be standalone mirrors, dielectric stacks, reflectivecoatings, or any other reflective material. Reflectors 1082 and 1084 maybe Fresnel reflectors.

Reflectors 1094 and 1096 may be used as beam diverting components toredirect light from light source 1072 toward the eye of the user.Reflectors 1094 and 1096 may be formed in a similar manner as describedabove with regard to FIG. 6. For example, reflectors 1094 and 1096 maybe formed as reflective coatings on prisms, and may be formed at thesame time that reflectors 1082 and 1084 are formed. Alternatively,reflectors 1094 and 1096 may be formed in any suitable manner, and maybe standalone mirrors, dielectric stacks, reflective coatings, or anyother reflective material. For example, reflectors 1094 and 1096 may bedeposited or diamond turned directly onto edge facets of reflectors 1082and 1084, respectively. Reflectors 1094 and 1096 may be angled toprovide an amount of tilt such that emitted light 1098 reaches the eyeof the user. As shown in FIG. 10C, reflector 1096 provides a greateramount of tilt than reflectors 1036 and 1066.

Light source 1072 may be positioned such that light 1090 is incident onreflector 1094, and is not incident on reflectors 1082 or 1084, therebypreventing stray light from being reflected toward mounting surface1088. Further, reflector 1082 prevents light from light source 1072 fromreaching the camera directly, thereby preventing noise in the capturedimages of the eye of the user. Light 1090 is reflected by reflector 1094toward reflector 1096, and is further reflected by reflector 1096 towardoutcoupling surface 1092.

The emitted light 1098 may be reflected by the eye of the user towardreflector 1084, which further reflects the light toward the camera.Reflector 1084 may be tilted at an angle such that the reflected lightreaches the camera. If multiple light sources and multiple correspondingreflective coatings are used, each reflective coating may be tilted at adifferent angle to reflect light from the eye of the user toward thecamera. Although reflector 1096 may be in the field of view of thecamera, reflector 1096 may be small enough that it will be nearlyinvisible to the camera. For example, reflector 1096 may only appear ina single pixel of the focal plane array of the camera. Reflector 1096may have a maximum linear dimension of 100 μm or less.

Reflectors 1082 and 1084 may be hot mirror coatings that reflect nearinfrared light and transmit visible light. Further, reflectors 1082 and1084 may be formed on a flat surface of prisms within substrate 1076. AFresnel lens may be formed on each flat surface in order to focus thereflected light. Alternatively, reflectors 1082 and 1084 may be formedon a curved surface of prisms within substrate 1076. The curved surfacemay include a concave surface, a convex surface, a cylindrical surface,an aspherical surface, and/or a freeform surface.

In other embodiments, the beam diverting components may be gratings,such as surface relief gratings or volume Bragg gratings. For example, asurface relief grating may be formed at an outcoupling surface of asubstrate, such as outcoupling surface 732 shown in FIG. 7.Alternatively, the gratings may be integrated within reflectors, such asreflectors 822 and/or 824 shown in FIG. 8A. In all embodiments, the beamdiverting components may include any combination of reflectors, prisms,lenses, and/or gratings.

FIG. 11 is a flow chart 1100 illustrating an example method of eyeillumination for eye tracking in a near-eye display, according tocertain embodiments. The method may be performed by, for example,eye-tracking unit 130 in near-eye display 120 of FIG. 1.

At block 1110, a light source (e.g., a VCSEL or a micro-LED) of aneye-tracking unit in a near-eye display device may emit light. The lightsource may be within a field of view of an eye of a user. In order toprevent the user from perceiving or being affected by the presence ofthe light source, a maximum dimension of the light source in a planeparallel to an emission surface of the light source may be less than 200μm. The light may be in the MR region of the spectrum, such as between830 nm and 860 nm, or between 930 nm and 980 nm. An angle of an emissioncone of the light from the light source may be less than 30°.

At block 1120, a beam diverting component may change the direction ofthe light from the light source in order to direct the light from thelight source toward the eye of the user. The beam diverting componentmay be designed such that at least a portion of the light is incident onthe eye of the user at an angle with respect to a vector that is normalto a surface of the eye of the user. Other portions of the light may beincident on the eye of the user at other angles, or may not be incidenton the eye of the user.

At block 1130, a reflector of the eye-tracking unit may receive lightthat is reflected by the eye of the user. The reflector may be withinthe field of view of the eye of the user. The light reflected by the eyeof the user may include light specularly reflected by the cornea of theeye and light diffusively reflected or diffracted by features within theeye, such as features on the iris or pupil of the eye.

At block 1140, a camera of the eye-tracking unit may receive light thatis reflected by the reflector. The light reflected by the eye of theuser may include light specularly reflected by the cornea of the eye andlight diffusively reflected or diffracted by features within the eye,such as features on the iris or pupil of the eye.

At block 1150, the camera may generate an image frame including an imageof the light source (a “glint”) reflected by the eye of the user, bydetecting the light reflected from the eye of the user using an imagesensor. In embodiments where multiple light sources are used, multipleglints may be captured in the captured image frame. The location(s) ofthe glint(s) in the captured image frame and/or other features in thecaptured image frame that correspond to features in different areas ofthe eye may then be used to determine a position of the user's eye asdescribed above.

Embodiments of the invention may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

FIG. 12 is a perspective view of an example near-eye display in the formof a head-mounted display (HMD) device 1200 for implementing some of theexample near-eye displays (e.g., near-eye display 120) disclosed herein.HMD device 1200 may be a part of, e.g., a virtual reality (VR) system,an augmented reality (AR) system, a mixed reality (MR) system, or somecombinations thereof. HMD device 1200 may include a body 1220 and a headstrap 1230. FIG. 12 shows a top side 1223, a front side 1225, and aright side 1227 of body 1220 in the perspective view. Head strap 1230may have an adjustable or extendible length. There may be a sufficientspace between body 1220 and head strap 1230 of HMD device 1200 forallowing a user to mount HMD device 1200 onto the user's head. Invarious embodiments, HMD device 1200 may include additional, fewer, ordifferent components. For example, in some embodiments, HMD device 1200may include eyeglass temples and temples tips as shown in, for example,FIGS. 2-4, rather than head strap 1230.

HMD device 1200 may present to a user media including virtual and/oraugmented views of a physical, real-world environment withcomputer-generated elements. Examples of the media presented by HMDdevice 1200 may include images (e.g., two-dimensional (2D) orthree-dimensional (3D) images), videos (e.g., 2D or 3D videos), audios,or some combinations thereof. The images and videos may be presented toeach eye of the user by one or more display assemblies (not shown inFIG. 12) enclosed in body 1220 of HMD device 1200. In variousembodiments, the one or more display assemblies may include a singleelectronic display panel or multiple electronic display panels (e.g.,one display panel for each eye of the user). Examples of the electronicdisplay panel(s) may include, for example, a liquid crystal display(LCD), an organic light emitting diode (OLED) display, an inorganiclight emitting diode (ILED) display, a micro-LED display, anactive-matrix organic light emitting diode (AMOLED) display, atransparent organic light emitting diode (TOLED) display, some otherdisplay, or some combinations thereof. HMD device 1200 may include twoeye box regions.

In some implementations, HMD device 1200 may include various sensors(not shown), such as depth sensors, motion sensors, position sensors,and eye tracking sensors. Some of these sensors may use a structuredlight pattern for sensing. In some implementations, HMD device 1200 mayinclude an input/output interface for communicating with a console. Insome implementations, HMD device 1200 may include a virtual realityengine (not shown) that can execute applications within HMD device 1200and receive depth information, position information, accelerationinformation, velocity information, predicted future positions, or somecombination thereof of HMD device 1200 from the various sensors. In someimplementations, the information received by the virtual reality enginemay be used for producing a signal (e.g., display instructions) to theone or more display assemblies. In some implementations, HMD device 1200may include locators (not shown, such as locators 126) located in fixedpositions on body 1220 relative to one another and relative to areference point. Each of the locators may emit light that is detectableby an external imaging device.

FIG. 13 is a simplified block diagram of an example electronic system1300 of an example near-eye display (e.g., HMD device) for implementingsome of the examples disclosed herein. Electronic system 1300 may beused as the electronic system of HMD device 1000 or other near-eyedisplays described above. In this example, electronic system 1300 mayinclude one or more processor(s) 1310 and a memory 1320. Processor(s)1310 may be configured to execute instructions for performing operationsat a number of components, and can be, for example, a general-purposeprocessor or microprocessor suitable for implementation within aportable electronic device. Processor(s) 1310 may be communicativelycoupled with a plurality of components within electronic system 1300. Torealize this communicative coupling, processor(s) 1310 may communicatewith the other illustrated components across a bus 1340. Bus 1340 may beany subsystem adapted to transfer data within electronic system 1300.Bus 1340 may include a plurality of computer buses and additionalcircuitry to transfer data.

Memory 1320 may be coupled to processor(s) 1310. In some embodiments,memory 1320 may offer both short-term and long-term storage and may bedivided into several units. Memory 1320 may be volatile, such as staticrandom access memory (SRAM) and/or dynamic random access memory (DRAM)and/or non-volatile, such as read-only memory (ROM), flash memory, andthe like. Furthermore, memory 1320 may include removable storagedevices, such as secure digital (SD) cards. Memory 1320 may providestorage of computer-readable instructions, data structures, programmodules, and other data for electronic system 1300. In some embodiments,memory 1320 may be distributed into different hardware modules. A set ofinstructions and/or code might be stored on memory 1320. Theinstructions might take the form of executable code that may beexecutable by electronic system 1300, and/or might take the form ofsource and/or installable code, which, upon compilation and/orinstallation on electronic system 1300 (e.g., using any of a variety ofgenerally available compilers, installation programs,compression/decompression utilities, etc.), may take the form ofexecutable code.

In some embodiments, memory 1320 may store a plurality of applicationmodules 1322 through 1324, which may include any number of applications.Examples of applications may include gaming applications, conferencingapplications, video playback applications, or other suitableapplications. The applications may include a depth sensing function oreye tracking function. Application modules 1322-1324 may includeparticular instructions to be executed by processor(s) 1310. In someembodiments, certain applications or parts of application modules1322-1324 may be executable by other hardware modules 1380. In certainembodiments, memory 1320 may additionally include secure memory, whichmay include additional security controls to prevent copying or otherunauthorized access to secure information.

In some embodiments, memory 1320 may include an operating system 1325loaded therein. Operating system 1325 may be operable to initiate theexecution of the instructions provided by application modules 1322-1324and/or manage other hardware modules 1380 as well as interfaces with awireless communication subsystem 1330 which may include one or morewireless transceivers. Operating system 1325 may be adapted to performother operations across the components of electronic system 1300including threading, resource management, data storage control and othersimilar functionality.

Wireless communication subsystem 1330 may include, for example, aninfrared communication device, a wireless communication device and/orchipset (such as a Bluetooth® device, an IEEE 802.11 device, a Wi-Fidevice, a WiMax device, cellular communication facilities, etc.), and/orsimilar communication interfaces. Electronic system 1300 may include oneor more antennas 1334 for wireless communication as part of wirelesscommunication subsystem 1330 or as a separate component coupled to anyportion of the system. Depending on desired functionality, wirelesscommunication subsystem 1330 may include separate transceivers tocommunicate with base transceiver stations and other wireless devicesand access points, which may include communicating with different datanetworks and/or network types, such as wireless wide-area networks(WWANs), wireless local area networks (WLANs), or wireless personal areanetworks (WPANs). A WWAN may be, for example, a WiMax (IEEE 802.16)network. A WLAN may be, for example, an IEEE 802.11x network. A WPAN maybe, for example, a Bluetooth network, an IEEE 802.15x, or some othertypes of network. The techniques described herein may also be used forany combination of WWAN, WLAN, and/or WPAN. Wireless communicationssubsystem 1330 may permit data to be exchanged with a network, othercomputer systems, and/or any other devices described herein. Wirelesscommunication subsystem 1330 may include a means for transmitting orreceiving data, such as identifiers of HMD devices, position data, ageographic map, a heat map, photos, or videos, using antenna(s) 1334 andwireless link(s) 1332. Wireless communication subsystem 1330,processor(s) 1310, and memory 1320 may together comprise at least a partof one or more of a means for performing some functions disclosedherein.

Embodiments of electronic system 1300 may also include one or moresensors 1390. Sensor(s) 1390 may include, for example, an image sensor,an accelerometer, a pressure sensor, a temperature sensor, a proximitysensor, a magnetometer, a gyroscope, an inertial sensor (e.g., a modulethat combines an accelerometer and a gyroscope), an ambient lightsensor, or any other similar module operable to provide sensory outputand/or receive sensory input, such as a depth sensor or a positionsensor. For example, in some implementations, sensor(s) 1390 may includeone or more inertial measurement units (IMUs) and/or one or moreposition sensors. An IMU may generate calibration data indicating anestimated position of the HMD device relative to an initial position ofthe HMD device, based on measurement signals received from one or moreof the position sensors. A position sensor may generate one or moremeasurement signals in response to motion of the HMD device. Examples ofthe position sensors may include, but are not limited to, one or moreaccelerometers, one or more gyroscopes, one or more magnetometers,another suitable type of sensor that detects motion, a type of sensorused for error correction of the IMU, or some combination thereof. Theposition sensors may be located external to the IMU, internal to theIMU, or some combination thereof. At least some sensors may use astructured light pattern for sensing.

Electronic system 1300 may include a display module 1360. Display module1360 may be a near-eye display, and may graphically present information,such as images, videos, and various instructions, from electronic system1300 to a user. Such information may be derived from one or moreapplication modules 1322-1324, virtual reality engine 1326, one or moreother hardware modules 1380, a combination thereof, or any othersuitable means for resolving graphical content for the user (e.g., byoperating system 1325). Display module 1360 may use liquid crystaldisplay (LCD) technology, light-emitting diode (LED) technology(including, for example, OLED, ILED, mLED, AMOLED, TOLED, etc.), lightemitting polymer display (LPD) technology, or some other displaytechnology.

Electronic system 1300 may include a user input/output module 1370. Userinput/output module 1370 may allow a user to send action requests toelectronic system 1300. An action request may be a request to perform aparticular action. For example, an action request may be to start or endan application or to perform a particular action within the application.User input/output module 1370 may include one or more input devices.Example input devices may include a touchscreen, a touch pad,microphone(s), button(s), dial(s), switch(es), a keyboard, a mouse, agame controller, or any other suitable device for receiving actionrequests and communicating the received action requests to electronicsystem 1300. In some embodiments, user input/output module 1370 mayprovide haptic feedback to the user in accordance with instructionsreceived from electronic system 1300. For example, the haptic feedbackmay be provided when an action request is received or has beenperformed.

Electronic system 1300 may include a camera 1350 that may be used totake photos or videos of a user, for example, for tracking the user'seye position. Camera 1350 may also be used to take photos or videos ofthe environment, for example, for VR, AR, or MR applications. Camera1350 may include, for example, a complementary metal-oxide-semiconductor(CMOS) image sensor with a few millions or tens of millions of pixels.In some implementations, camera 1350 may include two or more camerasthat may be used to capture 3-D images.

In some embodiments, electronic system 1300 may include a plurality ofother hardware modules 1380. Each of other hardware modules 1380 may bea physical module within electronic system 1300. While each of otherhardware modules 1380 may be permanently configured as a structure, someof other hardware modules 1380 may be temporarily configured to performspecific functions or temporarily activated. Examples of other hardwaremodules 1380 may include, for example, an audio output and/or inputmodule (e.g., a microphone or speaker), a near field communication (NFC)module, a rechargeable battery, a battery management system, awired/wireless battery charging system, etc. In some embodiments, one ormore functions of other hardware modules 1380 may be implemented insoftware.

In some embodiments, memory 1320 of electronic system 1300 may alsostore a virtual reality engine 1326. Virtual reality engine 1326 mayexecute applications within electronic system 1300 and receive positioninformation, acceleration information, velocity information, predictedfuture positions, or some combination thereof of the HMD device from thevarious sensors. In some embodiments, the information received byvirtual reality engine 1326 may be used for producing a signal (e.g.,display instructions) to display module 1360. For example, if thereceived information indicates that the user has looked to the left,virtual reality engine 1326 may generate content for the HMD device thatmirrors the user's movement in a virtual environment. Additionally,virtual reality engine 1326 may perform an action within an applicationin response to an action request received from user input/output module1370 and provide feedback to the user. The provided feedback may bevisual, audible, or haptic feedback. In some implementations,processor(s) 1310 may include one or more GPUs that may execute virtualreality engine 1326.

In various implementations, the above-described hardware and modules maybe implemented on a single device or on multiple devices that cancommunicate with one another using wired or wireless connections. Forexample, in some implementations, some components or modules, such asGPUs, virtual reality engine 1326, and applications (e.g., trackingapplication), may be implemented on a console separate from thehead-mounted display device. In some implementations, one console may beconnected to or support more than one HMD.

In alternative configurations, different and/or additional componentsmay be included in electronic system 1300. Similarly, functionality ofone or more of the components can be distributed among the components ina manner different from the manner described above. For example, in someembodiments, electronic system 1300 may be modified to include othersystem environments, such as an AR system environment and/or an MRenvironment.

The methods, systems, and devices discussed above are examples. Variousembodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods described may be performed in an order different from thatdescribed, and/or various stages may be added, omitted, and/or combined.Also, features described with respect to certain embodiments may becombined in various other embodiments. Different aspects and elements ofthe embodiments may be combined in a similar manner. Also, technologyevolves and, thus, many of the elements are examples that do not limitthe scope of the disclosure to those specific examples.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, embodiments may be practicedwithout these specific details. For example, well-known circuits,processes, systems, structures, and techniques have been shown withoutunnecessary detail in order to avoid obscuring the embodiments. Thisdescription provides example embodiments only, and is not intended tolimit the scope, applicability, or configuration of the invention.Rather, the preceding description of the embodiments will provide thoseskilled in the art with an enabling description for implementing variousembodiments. Various changes may be made in the function and arrangementof elements without departing from the spirit and scope of the presentdisclosure.

Also, some embodiments were described as processes depicted as flowdiagrams or block diagrams. Although each may describe the operations asa sequential process, many of the operations may be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Furthermore, embodiments of the methods may be implemented byhardware, software, firmware, middleware, microcode, hardwaredescription languages, or any combination thereof. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the associated tasks may be stored in acomputer-readable medium such as a storage medium. Processors mayperform the associated tasks.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized or special-purpose hardware might also be used,and/or particular elements might be implemented in hardware, software(including portable software, such as applets, etc.), or both. Further,connection to other computing devices such as network input/outputdevices may be employed.

With reference to the appended figures, components that can includememory can include non-transitory machine-readable media. The term“machine-readable medium” and “computer-readable medium,” as usedherein, refer to any storage medium that participates in providing datathat causes a machine to operate in a specific fashion. In embodimentsprovided hereinabove, various machine-readable media might be involvedin providing instructions/code to processing units and/or otherdevice(s) for execution. Additionally or alternatively, themachine-readable media might be used to store and/or carry suchinstructions/code. In many implementations, a computer-readable mediumis a physical and/or tangible storage medium. Such a medium may takemany forms, including, but not limited to, non-volatile media, volatilemedia, and transmission media. Common forms of computer-readable mediainclude, for example, magnetic and/or optical media such as compact disk(CD) or digital versatile disk (DVD), punch cards, paper tape, any otherphysical medium with patterns of holes, a RAM, a programmable read-onlymemory (PROM), an erasable programmable read-only memory (EPROM), aFLASH-EPROM, any other memory chip or cartridge, a carrier wave asdescribed hereinafter, or any other medium from which a computer canread instructions and/or code. A computer program product may includecode and/or machine-executable instructions that may represent aprocedure, a function, a subprogram, a program, a routine, anapplication (App), a subroutine, a module, a software package, a class,or any combination of instructions, data structures, or programstatements.

Those of skill in the art will appreciate that information and signalsused to communicate the messages described herein may be representedusing any of a variety of different technologies and techniques. Forexample, data, instructions, commands, information, signals, bits,symbols, and chips that may be referenced throughout the abovedescription may be represented by voltages, currents, electromagneticwaves, magnetic fields or particles, optical fields or particles, or anycombination thereof.

Terms, “and” and “or” as used herein, may include a variety of meaningsthat are also expected to depend at least in part upon the context inwhich such terms are used. Typically, “or” if used to associate a list,such as A, B, or C, is intended to mean A, B, and C, here used in theinclusive sense, as well as A, B, or C, here used in the exclusivesense. In addition, the term “one or more” as used herein may be used todescribe any feature, structure, or characteristic in the singular ormay be used to describe some combination of features, structures, orcharacteristics. However, it should be noted that this is merely anillustrative example and claimed subject matter is not limited to thisexample. Furthermore, the term “at least one of” if used to associate alist, such as A, B, or C, can be interpreted to mean any combination ofA, B, and/or C, such as A, AB, AC, BC, AA, ABC, AAB, AABBCCC, etc.

Further, while certain embodiments have been described using aparticular combination of hardware and software, it should be recognizedthat other combinations of hardware and software are also possible.Certain embodiments may be implemented only in hardware, or only insoftware, or using combinations thereof. In one example, software may beimplemented with a computer program product containing computer programcode or instructions executable by one or more processors for performingany or all of the steps, operations, or processes described in thisdisclosure, where the computer program may be stored on a non-transitorycomputer readable medium. The various processes described herein can beimplemented on the same processor or different processors in anycombination.

Where devices, systems, components or modules are described as beingconfigured to perform certain operations or functions, suchconfiguration can be accomplished, for example, by designing electroniccircuits to perform the operation, by programming programmableelectronic circuits (such as microprocessors) to perform the operationsuch as by executing computer instructions or code, or processors orcores programmed to execute code or instructions stored on anon-transitory memory medium, or any combination thereof. Processes cancommunicate using a variety of techniques, including, but not limitedto, conventional techniques for inter-process communications, anddifferent pairs of processes may use different techniques, or the samepair of processes may use different techniques at different times.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that additions, subtractions, deletions, and other modificationsand changes may be made thereunto without departing from the broaderspirit and scope as set forth in the claims. Thus, although specificembodiments have been described, these are not intended to be limiting.Various modifications and equivalents are within the scope of thefollowing claims.

What is claimed is:
 1. An illuminator for eye tracking comprising: alight source configured to be positioned within a field of view of aneye of a user; a beam diverting component configured to direct lightfrom the light source toward the eye of the user; a first reflectorconfigured to shadow the light source from a field of view of a camera;and a second reflector configured to receive light that is reflected bythe eye of the user, and to direct the light toward the camera such thatthe light is received by the camera.
 2. The illuminator of claim 1,wherein the first reflector is a first coating on a first prism, and thesecond reflector is a second coating on a second prism.
 3. Theilluminator of claim 2, wherein a first portion of the second prism thatis shadowed by the first reflector is uncoated, and a second portion ofthe second prism that is unshadowed by the first reflector is coated bythe first coating.
 4. The illuminator of claim 1, wherein the lightsource is configured to emit light that propagates between the firstreflector and the second reflector.
 5. The illuminator of claim 1,further comprising a substrate comprising a first surface on which thelight source is mounted and a second surface through which light isoutcoupled toward the eye of the user.
 6. The illuminator of claim 5,wherein the beam diverting component is formed on the second surface ofthe substrate, is indented toward the first surface of the substrate,and has a shape of a prism, a cone, a diffraction grating, or a lens. 7.The illuminator of claim 5, wherein the beam diverting component isformed on the second surface of the substrate, protrudes away from thefirst surface of the substrate, and has a shape of a prism or a cone. 8.The illuminator of claim 5, wherein: the beam diverting componentcomprises a third reflector and a fourth reflector, the third reflectoris configured to reflect light from the light source to the fourthreflector, and the fourth reflector is configured to reflect the lighttoward the second surface of the substrate.
 9. The illuminator of claim5, wherein the beam diverting component is a surface relief grating thatis formed at the second surface of the substrate or a volume Bragggrating.
 10. The illuminator of claim 1, wherein each of the firstreflector and the second reflector is configured to reflect infraredlight and to transmit visible light.
 11. A system for eye trackingcomprising: a light source configured to be positioned within a field ofview of an eye of a user; a beam diverting component configured todirect light from the light source toward the eye of the user, a cameraconfigured to capture an image of the light source reflected by the eyeof the user; a first reflector configured to shadow the light sourcefrom a field of view of the camera; and a second reflector configured toreceive light that is reflected by the eye of the user, and to directthe light toward the camera such that the light is received by thecamera.
 12. The system of claim 11, wherein the first reflector is afirst coating on a first prism, and the second reflector is a secondcoating on a second prism.
 13. The system of claim 12, wherein a firstportion of the second prism that is shadowed by the first reflector isuncoated, and a second portion of the second prism that is unshadowed bythe first reflector is coated by the first coating.
 14. The system ofclaim 11, wherein the light source is configured to emit light thatpropagates between the first reflector and the second reflector.
 15. Thesystem of claim 11, further comprising a substrate comprising a firstsurface on which the light source is mounted and a second surfacethrough which light is outcoupled toward the eye of the user.
 16. Thesystem of claim 15, wherein the beam diverting component is formed onthe second surface of the substrate, is indented toward the firstsurface of the substrate, and has a shape of a prism, a cone, adiffraction grating, or a lens.
 17. The system of claim 15, wherein thebeam diverting component is formed on the second surface of thesubstrate, protrudes away from the first surface of the substrate, andhas a shape of a prism or a cone.
 18. The system of claim 15, wherein:the beam diverting component comprises a third reflector and a fourthreflector, the third reflector is configured to reflect light from thelight source to the fourth reflector, and the fourth reflector isconfigured to reflect the light toward the second surface of thesubstrate.
 19. The system of claim 15, wherein the beam divertingcomponent is a surface relief grating that is formed at the secondsurface of the substrate or a volume Bragg grating.
 20. The system ofclaim 11, wherein each of the first reflector and the second reflectoris configured to reflect infrared light and to transmit visible light.